Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 929190 Details for
Bug 1070674
Provide informations about fencing in RHEV-M
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
21_08_14_engine.log (text/x-log), 2.49 MB, created by
sefi litmanovich
on 2014-08-21 12:34:07 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
sefi litmanovich
Created:
2014-08-21 12:34:07 UTC
Size:
2.49 MB
patch
obsolete
>2014-08-18 03:28:43,711 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) [17a6802f] Principal admin::internal synchronized >2014-08-18 03:28:46,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-08-18 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) Start deleteAgedOutAuditLogs >2014-08-18 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Start deleteAgedOutCommandEntities >2014-08-18 03:35:35,020 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Finished deleteAgedOutCommandEntities >2014-08-18 03:35:35,029 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) Finished deleteAgedOutAuditLogs >2014-08-18 04:28:43,716 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-08-18 04:28:46,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-18 05:28:43,728 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Principal admin::internal synchronized >2014-08-18 05:28:46,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-18 06:28:43,738 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-58) Principal admin::internal synchronized >2014-08-18 06:28:46,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [14e8b62f] Attempting to update VMs/Templates Ovf. >2014-08-18 07:28:43,743 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-08-18 07:28:46,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-08-18 08:28:43,749 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-08-18 08:28:46,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-08-18 09:28:43,756 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-08-18 09:28:46,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-18 10:28:43,767 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-08-18 10:28:46,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) [43c167db] Attempting to update VMs/Templates Ovf. >2014-08-18 11:15:09,914 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-08-18 11:15:09,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:15:10,014 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 11:15:50,121 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-6) ResourceManager::searchBusinessObjects - erroneous search text - Events:from 14441 >2014-08-18 11:19:15,958 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-18 11:19:15,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:19:16,021 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [1703f750] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:19:16,025 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [542c3aee] Running command: LogoutUserCommand internal: false. >2014-08-18 11:19:16,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [542c3aee] Correlation ID: 542c3aee, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:19:23,605 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 11:19:23,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:19:24,099 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,100 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,101 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_update_77ec04a0-e638-4cad] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:24,122 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,123 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:19:24,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:24,448 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:24,480 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:24,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 32fcc401 >2014-08-18 11:19:25,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 32fcc401 >2014-08-18 11:19:25,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: hosts_update_77ec04a0-e638-4cad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:19:29,272 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_1429d14e-d9f5-40e3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:19:29,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,388 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:29,394 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:29,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,500 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:29,534 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:29,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2ecab895 >2014-08-18 11:19:29,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2ecab895 >2014-08-18 11:19:29,842 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:29,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 63947ecf >2014-08-18 11:19:29,881 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, SetVdsStatusVDSCommand, log id: 63947ecf >2014-08-18 11:19:29,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:29,985 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:30,051 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:30,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7e658a1f >2014-08-18 11:19:30,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, FenceVdsVDSCommand, return: Done, log id: 7e658a1f >2014-08-18 11:19:30,258 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 11:19:35,260 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:19:35,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [6c0c45ba] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:35,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 64d51855 >2014-08-18 11:19:35,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 64d51855 >2014-08-18 11:19:35,614 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [6c0c45ba] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 11:19:35,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [6c0c45ba] Correlation ID: 6c0c45ba, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 11:19:35,674 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-3) [7002e688] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:35,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,825 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,830 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:19:35,862 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 73c6b6b5 >2014-08-18 11:19:35,867 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, SetVdsStatusVDSCommand, log id: 73c6b6b5 >2014-08-18 11:19:35,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:35,967 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:35,999 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:36,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4e6a6317 >2014-08-18 11:19:36,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Done, log id: 4e6a6317 >2014-08-18 11:19:36,249 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 11:19:41,250 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:19:41,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:19:41,324 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:19:41,356 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:19:41,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 24d716a7 >2014-08-18 11:19:41,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 24d716a7 >2014-08-18 11:19:41,632 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 11:19:41,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: 5d19c336, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 11:19:41,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_syncAction_1429d14e-d9f5-40e3, Job ID: a65cd8a6-854f-4915-8bc4-0a523c8162a4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 11:19:41,647 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:21:18,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-18 11:21:18,830 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 11:21:18,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:21:22,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 46379c02 >2014-08-18 11:21:22,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 46379c02 >2014-08-18 11:21:22,741 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [36bc2c13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:21:22,783 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [1d7cb8ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:21:22,809 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:21:22,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40f54bf7 >2014-08-18 11:21:23,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 40f54bf7 >2014-08-18 11:21:23,139 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 11:21:23,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 5d088cad >2014-08-18 11:21:23,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5d088cad >2014-08-18 11:21:23,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:21:23,668 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:21:23,700 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:21:23,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 68ca7585 >2014-08-18 11:21:24,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 68ca7585 >2014-08-18 11:21:24,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 293e7d1f >2014-08-18 11:21:24,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [7be8cee5] FINISH, SetMOMPolicyParametersVDSCommand, log id: 293e7d1f >2014-08-18 11:21:24,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 11:21:24,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7be8cee5] Correlation ID: 1d7cb8ec, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 11:21:24,641 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [2b16911e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:10,877 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,878 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,879 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [hosts_update_af885f2a-0c7f-4b17] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:10,887 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,888 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:10,959 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:10,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: hosts_update_af885f2a-0c7f-4b17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:27:32,013 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:32,017 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:32,043 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [17313e32] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:27:32,055 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [461d86e8] Running command: LogoutUserCommand internal: false. >2014-08-18 11:27:32,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [461d86e8] Correlation ID: 461d86e8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:27:48,742 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:48,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:48,761 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [47c80510] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:27:48,775 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [58ad6aba] Running command: LogoutUserCommand internal: false. >2014-08-18 11:27:48,793 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [58ad6aba] Correlation ID: 58ad6aba, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:27:53,973 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 11:27:53,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:27:54,346 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,347 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,348 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_26bdde25-1df1-4e3d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:54,366 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,367 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:27:54,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:54,590 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:54,622 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:54,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 69527d96 >2014-08-18 11:27:55,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 69527d96 >2014-08-18 11:27:55,338 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_update_26bdde25-1df1-4e3d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:27:59,024 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_6b3017e1-b5f7-4542] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:27:59,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,137 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,150 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:59,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,257 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,289 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:59,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 1b89f1ce >2014-08-18 11:27:59,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1b89f1ce >2014-08-18 11:27:59,601 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:27:59,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6995f951 >2014-08-18 11:27:59,640 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, SetVdsStatusVDSCommand, log id: 6995f951 >2014-08-18 11:27:59,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:27:59,744 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:27:59,777 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:27:59,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4cc9a80f >2014-08-18 11:27:59,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4cc9a80f >2014-08-18 11:28:00,025 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 11:28:05,027 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:28:05,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,100 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,132 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [33fa7b6e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:05,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6f2fe87d >2014-08-18 11:28:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6f2fe87d >2014-08-18 11:28:05,377 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [33fa7b6e] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 11:28:05,385 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [33fa7b6e] Correlation ID: 33fa7b6e, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 11:28:05,438 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-4) [c9aa456] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:28:05,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,613 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,619 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:28:05,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 53252745 >2014-08-18 11:28:05,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: 53252745 >2014-08-18 11:28:05,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:05,767 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:05,798 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:05,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3a2656ec >2014-08-18 11:28:05,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 3a2656ec >2014-08-18 11:28:06,007 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 11:28:11,008 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 11:28:11,079 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:28:11,081 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:28:11,115 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:28:11,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2784b300 >2014-08-18 11:28:11,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2784b300 >2014-08-18 11:28:11,366 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 11:28:11,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 27c919c9, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 11:28:11,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_6b3017e1-b5f7-4542, Job ID: 8a5b31a7-df6e-454a-883e-11df38c5a847, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 11:28:11,400 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 11:28:43,772 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-08-18 11:28:46,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-08-18 11:29:50,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-18 11:29:50,732 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 11:29:50,734 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:29:54,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 74535bf8 >2014-08-18 11:29:54,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GetHardwareInfoVDSCommand, log id: 74535bf8 >2014-08-18 11:29:54,541 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [747789c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:29:54,551 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [5034b2b4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:29:54,559 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 11:29:54,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d7aa421 >2014-08-18 11:29:54,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5d7aa421 >2014-08-18 11:29:54,902 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 11:29:54,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 597e00f7 >2014-08-18 11:29:55,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 597e00f7 >2014-08-18 11:29:55,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:29:55,267 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) [476c5856] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 11:29:55,301 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-86) [476c5856] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-18 11:29:55,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 598dfd59 >2014-08-18 11:29:55,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 598dfd59 >2014-08-18 11:29:55,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: aed5789 >2014-08-18 11:29:55,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [476c5856] FINISH, SetMOMPolicyParametersVDSCommand, log id: aed5789 >2014-08-18 11:29:55,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 11:29:55,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [476c5856] Correlation ID: 5034b2b4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 11:29:55,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [6e4aebb0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:36:17,051 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-5) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 11:39:12,641 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 11:39:12,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:39:12,649 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-3) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host rose07.qa.lab.tlv.redhat.com was restarted by admin >2014-08-18 11:39:12,661 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [330d397e] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:39:12,667 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7ee93c7e] Running command: LogoutUserCommand internal: false. >2014-08-18 11:39:12,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7ee93c7e] Correlation ID: 7ee93c7e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:39:20,200 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 11:39:20,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:39:22,580 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [f57d599] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:39:22,596 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [6d4cdfa8] Running command: LogoutUserCommand internal: false. >2014-08-18 11:39:22,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [6d4cdfa8] Correlation ID: 6d4cdfa8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 11:42:59,835 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,836 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,840 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_update_3403bf08-d2d3-4160] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 11:42:59,893 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,894 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 11:42:59,951 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 11:42:59,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: hosts_update_3403bf08-d2d3-4160, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 11:44:00,203 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 11:44:00,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 11:44:01,889 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [4510748b] Running command: LogoutBySessionCommand internal: false. >2014-08-18 11:44:01,893 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [727e3475] Running command: LogoutUserCommand internal: false. >2014-08-18 11:44:01,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [727e3475] Correlation ID: 727e3475, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 12:28:43,780 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-08-18 12:28:46,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-18 13:28:43,788 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-18 13:28:46,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-08-18 13:48:06,120 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 13:48:06,134 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:48:06,136 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-4) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:48:06,168 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-4) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 13:48:06,181 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [648e7fab] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:48:06,184 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [681e9e0c] Running command: LogoutUserCommand internal: false. >2014-08-18 13:48:06,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [681e9e0c] Correlation ID: 681e9e0c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:33,496 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:33,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:33,605 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [59af7713] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:33,608 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4a292ec6] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:33,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a292ec6] Correlation ID: 4a292ec6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:33,674 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:33,707 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [2a32b614] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:33,710 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [4428ee7b] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:33,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [4428ee7b] Correlation ID: 4428ee7b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:50,745 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:50,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:50,761 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-1) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:54:50,812 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) ResourceManager::searchBusinessObjects - erroneous search text - Events : Host monique-vds01.tlv.redhat.com from cluster >2014-08-18 13:54:50,830 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [275898fe] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:50,833 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [4a6dd6da] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:50,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4a6dd6da] Correlation ID: 4a6dd6da, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:54:59,234 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-18 13:54:59,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:54:59,248 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-11) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:54:59,279 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-11) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:54:59,283 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [2e3d38bb] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:54:59,287 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [ad65065] Running command: LogoutUserCommand internal: false. >2014-08-18 13:54:59,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [ad65065] Correlation ID: ad65065, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:55:07,814 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 13:55:07,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:55:07,819 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-9) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:55:07,852 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-9) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:55:07,856 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [45c2cc02] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:55:07,860 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [74d766b5] Running command: LogoutUserCommand internal: false. >2014-08-18 13:55:07,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [74d766b5] Correlation ID: 74d766b5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 13:55:16,506 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 13:55:16,512 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 13:55:16,513 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-3) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 13:55:16,553 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-3) ResourceManager::searchBusinessObjects - Invalid search text - Events : Host >2014-08-18 13:55:16,557 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [5bd24a23] Running command: LogoutBySessionCommand internal: false. >2014-08-18 13:55:16,561 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [50e560ed] Running command: LogoutUserCommand internal: false. >2014-08-18 13:55:16,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [50e560ed] Correlation ID: 50e560ed, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:07:52,847 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:07:52,858 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:07:54,855 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [39c28a66] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:07:54,858 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [3f9efa9b] Running command: LogoutUserCommand internal: false. >2014-08-18 14:07:54,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [3f9efa9b] Correlation ID: 3f9efa9b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:09:50,570 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:09:50,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:09:50,581 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-5) ResourceManager::searchBusinessObjects - erroneous search text - Events : sortby time 2 >2014-08-18 14:09:50,585 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [56be8e88] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:09:50,588 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [609465ad] Running command: LogoutUserCommand internal: false. >2014-08-18 14:09:50,596 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [609465ad] Correlation ID: 609465ad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:09:59,178 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:09:59,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:10:00,589 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [85567d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:10:00,595 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [79403ce8] Running command: LogoutUserCommand internal: false. >2014-08-18 14:10:00,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [79403ce8] Correlation ID: 79403ce8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:10:00,734 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 14:10:01,888 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [5d5af326] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:10:01,901 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [5fbe8d31] Running command: LogoutUserCommand internal: false. >2014-08-18 14:10:01,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5fbe8d31] Correlation ID: 5fbe8d31, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:12:31,943 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 14:12:31,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:12:31,963 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [478f3bc] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:12:31,966 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [821cbc4] Running command: LogoutUserCommand internal: false. >2014-08-18 14:12:31,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [821cbc4] Correlation ID: 821cbc4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:13:09,149 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 14:13:09,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:13:09,208 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [44be12d5] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:13:09,211 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [4e36eebd] Running command: LogoutUserCommand internal: false. >2014-08-18 14:13:09,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4e36eebd] Correlation ID: 4e36eebd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:13:47,511 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:13:47,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:13:47,573 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [2e6154c] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:13:47,576 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [44910958] Running command: LogoutUserCommand internal: false. >2014-08-18 14:13:47,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [44910958] Correlation ID: 44910958, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:23:07,889 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 14:23:07,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:23:07,927 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6551ab61] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:23:07,930 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [6da1a222] Running command: LogoutUserCommand internal: false. >2014-08-18 14:23:07,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [6da1a222] Correlation ID: 6da1a222, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:24:18,067 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:18,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:19,684 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [6c4485d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:24:19,687 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [859192e] Running command: LogoutUserCommand internal: false. >2014-08-18 14:24:19,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [859192e] Correlation ID: 859192e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:24:28,902 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:28,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:48,757 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-18 14:24:48,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:24:48,837 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [12c87c9a] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:24:48,840 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [1a2ac492] Running command: LogoutUserCommand internal: false. >2014-08-18 14:24:48,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [1a2ac492] Correlation ID: 1a2ac492, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:25:13,919 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-18 14:25:13,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:25:13,970 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [4dfcd826] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:25:13,984 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [2da4b11a] Running command: LogoutUserCommand internal: false. >2014-08-18 14:25:14,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [2da4b11a] Correlation ID: 2da4b11a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:25:28,920 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-18 14:25:28,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:26:07,524 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 14:26:07,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 14:26:07,529 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 14:26:07,587 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [272a238a] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:26:07,590 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [5ace3574] Running command: LogoutUserCommand internal: false. >2014-08-18 14:26:07,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [5ace3574] Correlation ID: 5ace3574, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:26:10,710 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-18 14:26:10,712 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-9) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 14:26:10,806 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2fc9a2d7] Running command: LogoutBySessionCommand internal: false. >2014-08-18 14:26:10,810 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [64e72e3e] Running command: LogoutUserCommand internal: false. >2014-08-18 14:26:10,814 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [64e72e3e] Correlation ID: 64e72e3e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 14:28:43,795 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-08-18 14:28:46,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-18 15:00:21,258 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 15:00:21,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:00:21,399 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [70b60610] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:00:21,402 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [148f6aad] Running command: LogoutUserCommand internal: false. >2014-08-18 15:00:21,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [148f6aad] Correlation ID: 148f6aad, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:01:56,228 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-18 15:01:56,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:01:56,497 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [63f795c4] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:01:56,511 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [378107a0] Running command: LogoutUserCommand internal: false. >2014-08-18 15:01:56,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [378107a0] Correlation ID: 378107a0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:08:14,837 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-18 15:08:14,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:08:14,849 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: ""2795"" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:430) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 15:08:14,962 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [6a8fc4] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:08:14,965 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [5c8108bd] Running command: LogoutUserCommand internal: false. >2014-08-18 15:08:14,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [5c8108bd] Correlation ID: 5c8108bd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:08:26,397 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 15:08:26,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 15:08:26,413 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "kaki" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:441) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.GeneratedMethodAccessor321.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 15:08:26,573 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [2fbe75db] Running command: LogoutBySessionCommand internal: false. >2014-08-18 15:08:26,590 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [51b8f278] Running command: LogoutUserCommand internal: false. >2014-08-18 15:08:26,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [51b8f278] Correlation ID: 51b8f278, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 15:28:43,803 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-18 15:28:46,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Attempting to update VMs/Templates Ovf. >2014-08-18 16:13:26,269 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-18 16:13:26,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 16:13:26,291 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1f320185] Running command: LogoutBySessionCommand internal: false. >2014-08-18 16:13:26,298 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3619c10e] Running command: LogoutUserCommand internal: false. >2014-08-18 16:13:26,301 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3619c10e] Correlation ID: 3619c10e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-18 16:13:34,405 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-18 16:13:34,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-18 16:13:34,721 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,722 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,723 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_66641311-7d6a-4efb] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:13:34,800 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:34,801 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:13:35,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:13:35,486 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:13:35,562 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:13:35,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6c98c2c9 >2014-08-18 16:14:15,865 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:15,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 6c98c2c9 >2014-08-18 16:14:15,869 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:14:16,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:14:16,145 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:14:16,146 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:14:16,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7f1e559e >2014-08-18 16:14:56,432 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:56,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 7f1e559e >2014-08-18 16:14:56,507 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:14:56,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_66641311-7d6a-4efb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:15:00,573 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [hosts_syncAction_a128507f-5f61-4ba3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:15:00,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:00,776 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:00,783 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:15:00,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:00,976 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:01,038 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:15:01,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 58f82e59 >2014-08-18 16:15:41,343 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:15:41,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 58f82e59 >2014-08-18 16:15:41,347 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:15:41,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:15:41,934 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:15:41,935 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:15:41,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 680c20af >2014-08-18 16:16:22,185 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:16:22,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 680c20af >2014-08-18 16:16:22,249 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:16:22,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 68588e4b >2014-08-18 16:16:22,477 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:16:22,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 68588e4b >2014-08-18 16:16:22,481 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 4290a94f-9e6e-4eed-a517-3401f613ebdf, trying another proxy... >2014-08-18 16:16:22,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:22,710 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:22,711 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:16:22,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 57f693c5 >2014-08-18 16:16:23,027 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:16:23,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 57f693c5 >2014-08-18 16:16:23,103 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:16:23,165 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 27f8a4f2 >2014-08-18 16:16:23,170 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, SetVdsStatusVDSCommand, log id: 27f8a4f2 >2014-08-18 16:16:23,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:23,361 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:23,426 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:16:23,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 87bf54f >2014-08-18 16:16:43,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Done, log id: 87bf54f >2014-08-18 16:16:43,675 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:16:48,677 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:16:48,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:16:48,821 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:16:48,888 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:16:48,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: a28473a >2014-08-18 16:17:29,173 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:17:29,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: a28473a >2014-08-18 16:17:29,177 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:17:29,481 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:17:29,483 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:17:29,484 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:17:29,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1c4e65dd >2014-08-18 16:18:09,730 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:18:09,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 1c4e65dd >2014-08-18 16:18:09,738 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Host rose07.qa.lab.tlv.redhat.com Primary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-18 16:18:09,818 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-18 16:18:09,822 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-18 16:18:10,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:10,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:10,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:10,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 4bec6c3e >2014-08-18 16:18:10,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4bec6c3e >2014-08-18 16:18:10,527 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:18:15,528 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:18:15,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:15,758 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:15,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:15,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 5c6f8788 >2014-08-18 16:18:16,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:18:16,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 5c6f8788 >2014-08-18 16:18:16,164 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:18:16,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:16,499 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:16,501 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) [697e2f3e] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-18 16:18:16,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 57e5e1da >2014-08-18 16:18:16,809 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:18:16,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 57e5e1da >2014-08-18 16:18:16,812 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Host rose07.qa.lab.tlv.redhat.com Secondary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-18 16:18:16,941 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-18 16:18:16,943 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-18 16:18:17,006 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to power fence host rose07.qa.lab.tlv.redhat.com. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >2014-08-18 16:18:17,014 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: 697e2f3e, Job ID: 8ea519be-710a-4a69-ba77-a20292347efa, Call Stack: null, Custom Event ID: -1, Message: Failed to stop Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-18 16:18:17,021 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Restart host action failed, updating host a8fdd580-c752-4537-9cae-e49e468db7da to NonResponsive >2014-08-18 16:18:17,089 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 25e8c28d >2014-08-18 16:18:17,095 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] FINISH, SetVdsStatusVDSCommand, log id: 25e8c28d >2014-08-18 16:18:17,103 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [697e2f3e] Correlation ID: hosts_syncAction_a128507f-5f61-4ba3, Job ID: 8ea519be-710a-4a69-ba77-a20292347efa, Call Stack: null, Custom Event ID: -1, Message: Failed to restart Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-18 16:18:17,108 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-10) [697e2f3e] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:18:17,109 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [] >2014-08-18 16:18:19,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6b1a9fe0 >2014-08-18 16:18:19,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, GetHardwareInfoVDSCommand, log id: 6b1a9fe0 >2014-08-18 16:18:19,560 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [5560aa4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:18:19,572 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-4) [2797c163] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:18:19,580 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:18:19,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13fc0608 >2014-08-18 16:18:19,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 13fc0608 >2014-08-18 16:18:19,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 16:18:19,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 15a5f06e >2014-08-18 16:18:19,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 15a5f06e >2014-08-18 16:18:20,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:18:20,165 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:18:20,227 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:18:20,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7baafda5 >2014-08-18 16:19:00,513 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:00,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 7baafda5 >2014-08-18 16:19:00,516 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:00,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:00,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:00,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-4) [32742e49] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-18 16:19:00,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 36de443b >2014-08-18 16:19:41,043 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:41,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 36de443b >2014-08-18 16:19:41,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 42fc0b0e >2014-08-18 16:19:41,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-4) [32742e49] FINISH, SetMOMPolicyParametersVDSCommand, log id: 42fc0b0e >2014-08-18 16:19:41,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 16:19:41,306 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-18 16:19:41,311 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [32742e49] Correlation ID: 2797c163, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com power management. >2014-08-18 16:19:41,384 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [21cb1557] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,144 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,146 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,148 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_update_b0005a08-f1fa-4eb7] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,155 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,156 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,234 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:50,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_update_b0005a08-f1fa-4eb7, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:50,416 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,417 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,418 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [hosts_update_15d66ad0-bf95-44a1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,430 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,431 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:50,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: hosts_update_15d66ad0-bf95-44a1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:50,816 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,817 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,818 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_7f90fdfa-3ced-4a96] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:50,902 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:50,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:19:51,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:51,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:51,437 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:51,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 566c6100 >2014-08-18 16:19:52,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 566c6100 >2014-08-18 16:19:52,266 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:52,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1bad30ac >2014-08-18 16:19:52,589 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:52,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1bad30ac >2014-08-18 16:19:52,592 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:52,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:52,917 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:52,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:52,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 66fe1988 >2014-08-18 16:19:53,148 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:53,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 66fe1988 >2014-08-18 16:19:53,226 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:53,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_7f90fdfa-3ced-4a96, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:19:56,323 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_8a31314d-362c-485e] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:19:56,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:56,527 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:56,533 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:56,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:56,894 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:57,030 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:57,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7718fd3f >2014-08-18 16:19:57,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7718fd3f >2014-08-18 16:19:57,403 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:57,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 48144f0d >2014-08-18 16:19:57,735 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:57,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 48144f0d >2014-08-18 16:19:57,739 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-18 16:19:58,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from data center DC1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:58,105 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds05.tlv.redhat.com from data center DC1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:58,106 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-18 16:19:58,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 25fadbc2 >2014-08-18 16:19:58,331 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-18 16:19:58,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 25fadbc2 >2014-08-18 16:19:58,486 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:19:58,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2156f0e7 >2014-08-18 16:19:58,560 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, SetVdsStatusVDSCommand, log id: 2156f0e7 >2014-08-18 16:19:58,773 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:19:58,776 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:19:58,992 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:19:58,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45b3771 >2014-08-18 16:19:59,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, FenceVdsVDSCommand, return: Done, log id: 45b3771 >2014-08-18 16:19:59,298 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-18 16:20:04,300 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:20:04,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:04,427 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:04,491 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) [3ff5a450] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:04,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6f6a4cb1 >2014-08-18 16:20:04,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6f6a4cb1 >2014-08-18 16:20:04,738 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [3ff5a450] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-18 16:20:04,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [3ff5a450] Correlation ID: 3ff5a450, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-18 16:20:04,833 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-4) [4b874bd2] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:20:05,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:05,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:05,125 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:20:05,187 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: db897b8 >2014-08-18 16:20:05,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: db897b8 >2014-08-18 16:20:05,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:05,383 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:05,448 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:05,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5b1d30ca >2014-08-18 16:20:05,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 5b1d30ca >2014-08-18 16:20:05,752 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-18 16:20:08,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:08,692 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:08,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:10,754 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-18 16:20:10,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:20:10,886 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:20:10,950 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:20:10,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1c4fcafe >2014-08-18 16:20:11,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1c4fcafe >2014-08-18 16:20:11,206 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-18 16:20:11,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 557029d9, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-18 16:20:11,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_8a31314d-362c-485e, Job ID: 27a4b2cc-6093-482a-b604-593eeeed49d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-18 16:20:11,223 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-18 16:20:13,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:13,794 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:13,795 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:18,904 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:18,906 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:18,907 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:24,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [79101bc3] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:24,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [79101bc3] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:24,021 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [79101bc3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:29,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:29,122 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:29,124 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:34,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:34,229 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:34,230 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:39,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:39,330 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:39,331 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:44,433 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:44,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:44,437 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:49,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:49,551 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:49,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:54,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:54,653 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:54,655 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:20:59,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [79101bc3] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:20:59,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [79101bc3] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:20:59,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [79101bc3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:04,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:04,879 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:04,898 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:10,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:10,034 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:10,041 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:15,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:15,140 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:15,164 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:20,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:20,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:20,289 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:25,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:25,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:25,398 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:30,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:30,498 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:30,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:35,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-18 16:21:35,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-18 16:21:35,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-18 16:21:39,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6cd4a494 >2014-08-18 16:21:39,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 6cd4a494 >2014-08-18 16:21:39,380 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [b7d0b91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:21:39,391 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [7ed347a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:21:39,399 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-18 16:21:39,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1ce0c747 >2014-08-18 16:21:39,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 1ce0c747 >2014-08-18 16:21:39,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-18 16:21:39,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 74808d86 >2014-08-18 16:21:39,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, ConnectStoragePoolVDSCommand, log id: 74808d86 >2014-08-18 16:21:40,157 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:21:40,159 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-18 16:21:40,221 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-18 16:21:40,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 688577da >2014-08-18 16:21:40,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 688577da >2014-08-18 16:21:40,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 55e7884 >2014-08-18 16:21:40,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [1e5179ab] FINISH, SetMOMPolicyParametersVDSCommand, log id: 55e7884 >2014-08-18 16:21:40,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-18 16:21:40,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [1e5179ab] Correlation ID: 7ed347a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-18 16:21:40,832 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [24033251] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,556 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,557 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,559 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_30e2bd93-e6b3-449d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,579 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,580 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:23:12,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_30e2bd93-e6b3-449d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:23:12,819 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,820 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,821 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-11) [hosts_update_05738963-a7ff-41b8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-18 16:23:12,858 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,859 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-11) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-18 16:23:12,911 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-18 16:23:12,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: hosts_update_05738963-a7ff-41b8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-18 16:28:43,809 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-08-18 16:28:46,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-08-18 17:28:43,818 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-08-18 17:28:46,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-08-18 18:28:43,828 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) [79101bc3] Principal admin::internal synchronized >2014-08-18 18:28:46,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-08-18 19:28:43,835 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-18 19:28:46,430 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-18 20:28:43,842 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-08-18 20:28:46,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-08-18 21:28:43,853 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-08-18 21:28:46,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-08-18 22:28:43,863 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-08-18 22:28:46,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-08-18 23:28:43,868 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-18 23:28:46,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-08-19 00:28:43,879 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) [43c167db] Principal admin::internal synchronized >2014-08-19 00:28:46,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) [6e4aebb0] Attempting to update VMs/Templates Ovf. >2014-08-19 01:28:43,884 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-08-19 01:28:46,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Attempting to update VMs/Templates Ovf. >2014-08-19 02:28:43,890 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-08-19 02:28:46,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [5b09d125] Attempting to update VMs/Templates Ovf. >2014-08-19 03:28:43,894 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized >2014-08-19 03:28:46,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-08-19 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Start deleteAgedOutCommandEntities >2014-08-19 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Start deleteAgedOutAuditLogs >2014-08-19 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Finished deleteAgedOutAuditLogs >2014-08-19 03:35:35,017 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Finished deleteAgedOutCommandEntities >2014-08-19 04:28:43,906 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) [2bbfee0d] Principal admin::internal synchronized >2014-08-19 04:28:46,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-19 05:28:43,920 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) [14e8b62f] Principal admin::internal synchronized >2014-08-19 05:28:46,662 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-08-19 06:28:43,940 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-19 06:28:46,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [2086a3b3] Attempting to update VMs/Templates Ovf. >2014-08-19 07:28:43,950 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-19 07:28:46,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) [24033251] Attempting to update VMs/Templates Ovf. >2014-08-19 08:28:43,955 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-08-19 08:28:46,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-08-19 09:28:43,960 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-50) Principal admin::internal synchronized >2014-08-19 09:28:46,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-08-19 10:18:07,078 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:28:43,967 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-19 10:28:46,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-08-19 10:30:52,102 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginAdminUserCommand internal: false. >2014-08-19 10:30:52,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:30:52,201 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:14,066 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:14,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:33:14,109 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2c8106d3] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:33:14,115 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [34ddc70e] Running command: LogoutUserCommand internal: false. >2014-08-19 10:33:14,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [34ddc70e] Correlation ID: 34ddc70e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:33:27,794 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false. >2014-08-19 10:33:27,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:33:27,852 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [52790aa] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:33:27,856 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1b306f84] Running command: LogoutUserCommand internal: false. >2014-08-19 10:33:27,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1b306f84] Correlation ID: 1b306f84, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:34:20,602 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-19 10:34:20,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:34:24,996 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [2ed4b4cc] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:34:25,003 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [51f1b52b] Running command: LogoutUserCommand internal: false. >2014-08-19 10:34:25,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [51f1b52b] Correlation ID: 51f1b52b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:34:40,151 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-19 10:34:40,187 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 10:34:40,248 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,249 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,250 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-10) [53259bd7] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:34:40,256 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,257 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:34:40,321 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:34:40,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: 53259bd7, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:34:40,348 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [1e15cdc7] Running command: LogoutBySessionCommand internal: false. >2014-08-19 10:34:40,351 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [2ad755fa] Running command: LogoutUserCommand internal: false. >2014-08-19 10:34:40,361 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2ad755fa] Correlation ID: 2ad755fa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 10:45:10,322 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-5) [3bb05623] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-19 10:45:10,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [3bb05623] Correlation ID: 3bb05623, Call Stack: null, Custom Event ID: -1, Message: Data Center DC2, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-08-19 10:45:32,449 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-9) [4b65fcb9] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-19 10:45:32,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [4b65fcb9] Correlation ID: 4b65fcb9, Call Stack: null, Custom Event ID: -1, Message: Cluster Cluster_21 was added by admin >2014-08-19 10:45:49,145 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [42547f73] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: POOL >] >2014-08-19 10:45:49,203 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:49,208 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 57002d35 >2014-08-19 10:45:49,210 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] VDS monique-vds05.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-19 10:45:49,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, ignoreStopFailed = false), log id: 2a83f716 >2014-08-19 10:45:49,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] START, SpmStopVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 24f14f1e >2014-08-19 10:45:49,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] SpmStopVDSCommand::Stopping SPM on vds monique-vds05.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:45:49,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, SpmStopVDSCommand, log id: 24f14f1e >2014-08-19 10:45:49,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, ResetIrsVDSCommand, log id: 2a83f716 >2014-08-19 10:45:49,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] FINISH, SetVdsStatusVDSCommand, log id: 57002d35 >2014-08-19 10:45:49,481 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: ab2440ae-4286-4465-beed-c2ccc8256f8b value: POOL >] >2014-08-19 10:45:49,537 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-8) [42547f73] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:49,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [42547f73] Correlation ID: 42547f73, Job ID: 3a00219a-580c-41b8-add3-4df882d3afa9, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-19 10:45:51,324 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com >2014-08-19 10:45:51,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: monique-vds05.tlv.redhat.com. >2014-08-19 10:45:51,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Removing vds [4290a94f-9e6e-4eed-a517-3401f613ebdf] from the domain in maintenance cache >2014-08-19 10:45:51,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Removing host(s) [4290a94f-9e6e-4eed-a517-3401f613ebdf] from hosts unseen domain report cache >2014-08-19 10:45:51,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 3), log id: 11437dcd >2014-08-19 10:45:56,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, DisconnectStoragePoolVDSCommand, log id: 11437dcd >2014-08-19 10:45:56,415 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:45:56,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 507ef248 >2014-08-19 10:45:56,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [41fc2ff5] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 507ef248 >2014-08-19 10:45:57,076 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:45:57,079 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 10:45:57,087 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Correlation ID: 1e0d0acf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 10:45:57,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 10:45:57,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-19 10:45:57,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d5dcc45 >2014-08-19 10:45:57,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling started: taskId = 32c652db-c90e-4c68-8fd4-f20b116e2939 >2014-08-19 10:45:58,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling ended: taskId = 32c652db-c90e-4c68-8fd4-f20b116e2939 task status = finished >2014-08-19 10:45:58,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] spmStart polling ended, spm status: SPM >2014-08-19 10:45:58,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=32c652db-c90e-4c68-8fd4-f20b116e2939), log id: 64c66230 >2014-08-19 10:45:58,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] FINISH, HSMClearTaskVDSCommand, log id: 64c66230 >2014-08-19 10:45:58,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dca6086, log id: 4d5dcc45 >2014-08-19 10:45:58,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-19 10:45:58,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [1e0d0acf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-19 10:45:58,321 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-19) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:45:58,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 7b750113 >2014-08-19 10:45:58,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:45:58,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 31a3ddc5 >2014-08-19 10:45:58,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31a3ddc5 >2014-08-19 10:45:58,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7b750113 >2014-08-19 10:45:58,558 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-19) [1e0d0acf] Discovered no tasks on Storage Pool DC1 >2014-08-19 10:45:59,275 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS, ID: 682324a4-a272-4d9d-a816-ef3a35d4270f Type: VdsGroups >2014-08-19 10:45:59,288 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] START, RemoveVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 2e022485 >2014-08-19 10:45:59,289 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] vdsManager::disposing >2014-08-19 10:45:59,291 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] FINISH, RemoveVdsVDSCommand, log id: 2e022485 >2014-08-19 10:45:59,295 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] START, AddVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 70d53665 >2014-08-19 10:45:59,296 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] AddVds - entered , starting logic to add VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf >2014-08-19 10:45:59,298 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] AddVds - VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added, will try to add it to the resource manager >2014-08-19 10:45:59,299 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] Entered VdsManager constructor >2014-08-19 10:45:59,312 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-11) [53ddd85b] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-19 10:45:59,313 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-11) [53ddd85b] VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-19 10:45:59,314 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-11) [53ddd85b] FINISH, AddVdsVDSCommand, log id: 70d53665 >2014-08-19 10:45:59,343 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-11) [43d7cdf3] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,373 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-11) [43d7cdf3] Lock freed to object EngineLock [exclusiveLocks= key: 5d41097a-d767-4fa2-baf4-eedd5526f920 value: REGISTER_VDS >, sharedLocks= ] >2014-08-19 10:45:59,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [43d7cdf3] Correlation ID: 53ddd85b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:45:59,529 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,647 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,648 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-4) [2e528bd6] No changes were detected in setup networks for host monique-vds05.tlv.redhat.com (ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf) >2014-08-19 10:45:59,650 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,650 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,651 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [1f0718dd] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:45:59,664 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,665 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:45:59,670 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1f0718dd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-19 10:45:59,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1f0718dd] Correlation ID: 1f0718dd, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:46:02,321 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-19 10:46:03,953 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8) [509fbafa] Lock Acquired to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS >, sharedLocks= ] >2014-08-19 10:46:03,986 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:46:03,988 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Before acquiring lock in order to prevent monitoring for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:03,989 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Lock acquired, from now a monitoring of host will be skipped for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:04,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 72da14bf >2014-08-19 10:46:04,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] FINISH, SetVdsStatusVDSCommand, log id: 72da14bf >2014-08-19 10:46:04,055 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Activate finished. Lock released. Monitoring can run now for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-08-19 10:46:04,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Correlation ID: 509fbafa, Job ID: 74ba3d5a-9eb3-49f4-9dd5-1b5d143d0b02, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was activated by admin. >2014-08-19 10:46:04,065 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [509fbafa] Lock freed to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS >, sharedLocks= ] >2014-08-19 10:46:05,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 57ca2011 >2014-08-19 10:46:05,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 57ca2011 >2014-08-19 10:46:05,776 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:46:05,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 60976a90 >2014-08-19 10:46:05,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [7f0a6f2a] FINISH, GetHardwareInfoVDSCommand, log id: 60976a90 >2014-08-19 10:46:05,967 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-19 10:46:05,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 3c9deae3 >2014-08-19 10:46:06,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [22f00d97] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3c9deae3 >2014-08-19 10:46:06,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [22f00d97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-08-19 10:46:06,112 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [22f00d97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-19 10:46:06,145 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [615e8cb8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 10:47:54,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:47:54,988 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-19 10:47:54,991 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:47:54,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 14ff0746 >2014-08-19 10:47:55,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 14ff0746 >2014-08-19 10:48:01,113 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,114 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,116 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-12) [593c4211] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:48:01,135 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,136 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:01,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:01,371 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:48:01,403 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:01,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1ff0b870 >2014-08-19 10:48:01,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1ff0b870 >2014-08-19 10:48:01,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: 593c4211, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:48:20,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:20,741 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 10:48:20,775 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:20,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4bcc4373 >2014-08-19 10:48:21,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4bcc4373 >2014-08-19 10:48:28,882 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,883 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,884 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-9) [22c94d72] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:48:28,902 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:28,903 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:48:29,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:48:29,215 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:48:29,248 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:48:29,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 368f3820 >2014-08-19 10:48:29,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 368f3820 >2014-08-19 10:48:29,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: 22c94d72, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 10:52:38,039 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-12) [4b34bd4c] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-19 10:52:38,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:38,182 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:38,202 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:38,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:38,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:38,444 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:38,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 64a23697 >2014-08-19 10:52:39,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [4b34bd4c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 64a23697 >2014-08-19 10:52:39,341 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:39,386 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 34dc00da >2014-08-19 10:52:39,387 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-19 10:52:39,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 1282d046 >2014-08-19 10:52:39,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 68927cce >2014-08-19 10:52:39,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:52:39,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, SpmStopVDSCommand, log id: 68927cce >2014-08-19 10:52:39,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, ResetIrsVDSCommand, log id: 1282d046 >2014-08-19 10:52:39,535 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, SetVdsStatusVDSCommand, log id: 34dc00da >2014-08-19 10:52:39,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:39,683 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:39,715 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:39,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 51a3396b >2014-08-19 10:52:39,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, FenceVdsVDSCommand, return: Done, log id: 51a3396b >2014-08-19 10:52:39,999 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-19 10:52:41,843 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:52:41,851 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 10:52:41,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Correlation ID: 1354ecb5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 10:52:42,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 10:52:42,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-19 10:52:42,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e2db11f >2014-08-19 10:52:42,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling started: taskId = 45e76b22-aeca-404a-8b68-12c36537d155 >2014-08-19 10:52:43,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling ended: taskId = 45e76b22-aeca-404a-8b68-12c36537d155 task status = finished >2014-08-19 10:52:43,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] spmStart polling ended, spm status: SPM >2014-08-19 10:52:43,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=45e76b22-aeca-404a-8b68-12c36537d155), log id: 8779f00 >2014-08-19 10:52:43,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] FINISH, HSMClearTaskVDSCommand, log id: 8779f00 >2014-08-19 10:52:43,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) [1354ecb5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36b4d6dc, log id: 3e2db11f >2014-08-19 10:52:43,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-19 10:52:43,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1354ecb5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-19 10:52:43,224 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-45) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-19 10:52:43,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 5e56d350 >2014-08-19 10:52:43,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 10:52:43,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 786eee8b >2014-08-19 10:52:43,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 786eee8b >2014-08-19 10:52:43,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e56d350 >2014-08-19 10:52:43,385 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-45) [1354ecb5] Discovered no tasks on Storage Pool DC1 >2014-08-19 10:52:45,000 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-19 10:52:45,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,075 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,113 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:45,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 6af5134d >2014-08-19 10:52:45,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6af5134d >2014-08-19 10:52:45,415 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-19 10:52:45,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [6f6bace1] Correlation ID: 6f6bace1, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-19 10:52:45,501 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-46) [663ff72b] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:45,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,788 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,794 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:52:45,826 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 541ed097 >2014-08-19 10:52:45,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SetVdsStatusVDSCommand, log id: 541ed097 >2014-08-19 10:52:45,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:45,942 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:45,974 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:45,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5e005195 >2014-08-19 10:52:46,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, FenceVdsVDSCommand, return: Done, log id: 5e005195 >2014-08-19 10:52:46,325 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-19 10:52:51,327 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-19 10:52:51,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:52:51,421 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:52:51,456 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-46) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:52:51,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 668d50b6 >2014-08-19 10:52:51,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 668d50b6 >2014-08-19 10:52:51,724 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-46) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-19 10:52:51,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: 4bc73fe3, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-19 10:52:51,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) Correlation ID: 4b34bd4c, Job ID: c872a9c9-b74e-41ae-86ac-18a0e428d02a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-19 10:52:51,744 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-8-thread-46) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-19 10:54:31,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-19 10:54:31,080 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 10:54:31,081 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 10:54:34,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 504c4e84 >2014-08-19 10:54:35,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, GetHardwareInfoVDSCommand, log id: 504c4e84 >2014-08-19 10:54:35,070 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [28f19b5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:54:35,081 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-76) [6c272a9b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:54:35,089 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 10:54:35,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c0f9072 >2014-08-19 10:54:35,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4c0f9072 >2014-08-19 10:54:35,364 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-19 10:54:35,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 11db6525 >2014-08-19 10:54:35,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 11db6525 >2014-08-19 10:54:35,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:54:35,701 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) [6898462d] Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:54:35,749 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-76) [6898462d] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:54:35,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3dfa9058 >2014-08-19 10:54:36,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3dfa9058 >2014-08-19 10:54:36,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 69ced075 >2014-08-19 10:54:36,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-76) [6898462d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 69ced075 >2014-08-19 10:54:36,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-19 10:54:36,152 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [6898462d] Correlation ID: 6c272a9b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-19 10:54:36,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [3e33609] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:59:02,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:59:02,363 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 10:59:02,395 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:59:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 445f6092 >2014-08-19 10:59:03,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 445f6092 >2014-08-19 10:59:08,781 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,783 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,784 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [4b38753f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 10:59:08,878 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:08,879 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 10:59:09,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 10:59:09,407 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 10:59:09,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 10:59:09,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 763c71a2 >2014-08-19 10:59:09,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 763c71a2 >2014-08-19 10:59:09,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 4b38753f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 11:06:33,112 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-19 11:06:33,114 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-19 11:06:33,114 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-19 11:06:33,115 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-19 11:06:33,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-19 11:06:33,178 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-19 11:06:33,180 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-19 11:06:33,181 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-19 11:06:33,181 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-19 11:06:33,182 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-19 11:06:33,183 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-19 11:06:33,184 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-19 11:06:33,185 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-19 11:06:33,186 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-19 11:06:33,187 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-19 11:06:33,187 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-19 11:06:33,188 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-19 11:06:33,201 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-19 11:06:33,202 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-19 11:06:33,203 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-19 11:06:33,204 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-19 11:06:33,205 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-19 11:06:33,206 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-08-19 11:06:33,206 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-19 11:06:33,207 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-19 11:06:33,208 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-19 11:06:33,209 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-19 11:06:33,210 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-19 11:06:33,211 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-19 11:06:33,214 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-19 11:06:33,215 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-19 11:06:33,216 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-19 11:06:33,216 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-19 11:06:33,217 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-19 11:06:33,218 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-19 11:06:33,219 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-19 11:06:33,219 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-19 11:06:33,220 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-19 11:06:33,221 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-19 11:06:33,222 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-19 11:06:33,223 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-19 11:06:33,223 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-19 11:06:33,224 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-19 11:06:33,246 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-19 11:06:33,247 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-19 11:06:33,248 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-19 11:06:33,249 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-19 11:06:33,250 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-19 11:06:33,250 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-19 11:06:33,251 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-19 11:06:33,252 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-19 11:06:33,253 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-19 11:06:33,254 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-19 11:06:33,254 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-19 11:06:33,255 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-19 11:06:33,256 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-19 11:06:33,269 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-19 11:06:33,270 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-19 11:06:33,270 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-19 11:06:33,271 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-19 11:06:33,272 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-19 11:06:33,273 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-08-19 11:06:33,273 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-19 11:06:33,274 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-19 11:06:33,275 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-19 11:06:33,275 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-19 11:06:33,276 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-19 11:06:33,277 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-19 11:06:35,852 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-19 11:06:37,268 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-19 11:06:37,269 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-19 11:06:37,603 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-19 11:06:37,604 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-19 11:06:37,609 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-19 11:06:37,610 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-19 11:06:37,611 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-19 11:06:37,694 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-19 11:06:37,695 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-19 11:06:37,696 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-19 11:06:37,699 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-19 11:06:37,702 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-19 11:06:37,728 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-19 11:06:37,773 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-19 11:06:37,774 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-19 11:06:37,800 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-19 11:06:38,076 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-19 11:06:38,163 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-19 11:06:38,206 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-19 11:06:38,209 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-19 11:06:38,209 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-19 11:06:38,210 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-19 11:06:38,211 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-19 11:06:38,356 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-19 11:06:38,358 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-19 11:06:38,359 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-19 11:06:38,361 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-19 11:06:38,364 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-19 11:06:38,378 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-19 11:06:38,539 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-19 11:06:38,540 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-19 11:06:38,542 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-19 11:06:38,543 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-19 11:06:39,008 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-19 11:06:39,010 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-19 11:06:39,040 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-19 11:06:39,043 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-19 11:06:39,371 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:39,468 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-19 11:06:39,653 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-19 11:06:39,979 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-19 11:06:39,980 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:40,013 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-19 11:06:40,052 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-19 11:06:40,053 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-19 11:06:40,083 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-19 11:06:40,109 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-19 11:06:40,171 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-19 11:06:40,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-19 11:06:40,193 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-19 11:06:40,194 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-19 11:06:40,194 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-19 11:06:40,195 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-19 11:06:40,238 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-19 11:06:40,243 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-19 11:06:40,245 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-19 11:06:40,260 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-19 11:06:40,296 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-19 11:06:40,373 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-19 11:06:40,374 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-19 11:06:40,375 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-19 11:06:40,377 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-19 11:06:45,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 4a2fc64b >2014-08-19 11:06:45,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 5c48835c >2014-08-19 11:06:45,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 40404cd5 >2014-08-19 11:06:45,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 5c48835c >2014-08-19 11:06:45,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 4a2fc64b >2014-08-19 11:06:45,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 40404cd5 >2014-08-19 11:06:46,889 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-08-19 11:06:46,885 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-08-19 11:06:46,943 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-19 11:06:47,486 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [434e9f98] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-19 11:06:47,482 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [2962f4d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:06:47,486 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 11:06:50,465 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [1307f0db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 11:06:50,499 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [1307f0db] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 11:06:50,599 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-19 11:06:50,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: UNASSIGNED not exist in string table >2014-08-19 11:06:50,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-19 11:06:50,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-19 11:06:50,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-19 11:06:50,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-19 11:06:50,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-19 11:06:50,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-19 11:06:50,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-19 11:06:50,650 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-19 11:06:50,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-19 11:06:50,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_STOPPED not exist in string table >2014-08-19 11:06:50,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_STARTED not exist in string table >2014-08-19 11:06:50,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: DWH_ERROR not exist in string table >2014-08-19 11:06:50,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-19 11:06:50,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-19 11:06:50,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-19 11:06:50,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-19 11:06:51,992 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Correlation ID: 1307f0db, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-19 11:06:52,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1307f0db] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-08-19 11:06:52,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1307f0db] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-19 11:06:52,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1307f0db] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-19 11:06:52,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 49a9e9c3 >2014-08-19 11:06:53,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-19 11:06:53,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 7651d1d8 >2014-08-19 11:06:53,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7651d1d8 >2014-08-19 11:06:53,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [1307f0db] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49a9e9c3 >2014-08-19 11:06:53,163 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [1307f0db] Discovered no tasks on Storage Pool DC1 >2014-08-19 11:06:54,497 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [32e7cf13] Running command: LogoutUserCommand internal: false. >2014-08-19 11:06:54,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [32e7cf13] Correlation ID: 32e7cf13, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-19 11:07:01,915 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-19 11:07:01,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 11:07:42,805 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-08-19 11:07:42,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-19 11:09:44,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:09:44,439 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:09:44,482 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-9) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:09:44,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45fce454 >2014-08-19 11:09:45,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 45fce454 >2014-08-19 11:09:48,385 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,386 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,388 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-12) [3f945f1f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:09:48,767 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:48,768 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-12) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:09:50,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:09:50,307 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:09:50,393 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-12) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:09:50,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 71f9fc3f >2014-08-19 11:09:50,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-12) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 71f9fc3f >2014-08-19 11:09:50,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: 3f945f1f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 11:37:01,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:01,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds01.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:37:01,756 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:01,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: b3c2b77 >2014-08-19 11:37:42,062 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-19 11:37:42,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: b3c2b77 >2014-08-19 11:37:42,066 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-19 11:37:42,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:42,211 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Using Host monique-vds05.tlv.redhat.com from other data center as proxy to execute Status command on Host >2014-08-19 11:37:42,212 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-10) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:42,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 1ef9da25 >2014-08-19 11:37:42,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-10) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1ef9da25 >2014-08-19 11:37:52,741 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,754 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,763 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [7dcdbadc] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-19 11:37:52,793 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:52,794 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-19 11:37:53,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:37:53,079 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:37:53,111 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:37:53,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3f1fad32 >2014-08-19 11:38:33,551 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-19 11:38:33,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:rose07-mgmt.qa.lab.tlv.redhat.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 3f1fad32 >2014-08-19 11:38:33,556 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-19 11:38:33,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from other data center was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-19 11:38:33,926 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds05.tlv.redhat.com from other data center as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-19 11:38:33,927 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-19 11:38:33,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3753356b >2014-08-19 11:38:34,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3753356b >2014-08-19 11:38:34,142 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 7dcdbadc, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-19 12:06:39,711 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-19 12:06:40,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-08-19 12:12:16,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-30) Command ListVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:16,322 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:16,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-19 12:12:16,348 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-19 12:12:16,457 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:16,459 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:16,460 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-19 12:12:16,476 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-19 12:12:21,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-19 12:12:21,091 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:12:21,096 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [45b4b475] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-19 12:12:21,113 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [45b4b475] Correlation ID: 45b4b475, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-19 12:12:21,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,371 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:21,372 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:21,406 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,505 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:21,507 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:21,508 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:21,530 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,421 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:26,502 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Server failed to respond, vds_id = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds_name = monique-vds05.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,512 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com is non responsive. >2014-08-19 12:12:26,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,546 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:26,550 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-35) ResourceManager::vdsNotResponding entered for Host 4290a94f-9e6e-4eed-a517-3401f613ebdf, monique-vds05.tlv.redhat.com >2014-08-19 12:12:26,604 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:26,715 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Server failed to respond, vds_id = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:26,737 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-19 12:12:26,742 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] Opening SSH Soft Fencing session on host monique-vds05.tlv.redhat.com >2014-08-19 12:12:26,756 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-08-19 12:12:26,786 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-3) ResourceManager::vdsNotResponding entered for Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, monique-vds01.tlv.redhat.com >2014-08-19 12:12:26,820 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:26,889 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-19 12:12:26,903 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-08-19 12:12:31,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:31,706 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:31,713 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:31,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:31,888 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:31,903 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:36,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:36,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:36,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:36,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:36,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:36,944 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:41,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:41,814 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:41,821 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:41,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:41,988 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:42,010 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:46,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:46,869 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:46,902 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:47,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:47,053 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:47,059 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:51,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:51,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:51,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:52,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:52,107 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:52,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:57,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:57,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:57,057 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:12:57,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:12:57,161 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:12:57,167 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:02,096 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:02,098 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:02,104 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:02,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:02,207 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:02,220 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:07,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:07,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:07,166 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:07,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:07,254 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:07,302 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:12,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:12,204 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:12,213 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:12,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:12,349 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:12,359 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:17,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:17,256 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:17,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:17,397 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:17,399 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:17,429 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:22,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:22,313 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:22,321 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:22,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:22,462 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:22,472 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:27,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:27,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:27,369 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:27,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:27,517 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:27,523 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:29,989 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-35) [61b9d0bb] SSH Soft Fencing command failed on host monique-vds05.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view6.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-19 12:13:30,003 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-3) [75f680d5] SSH Soft Fencing command failed on host monique-vds01.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view6.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-19 12:13:30,152 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [47df259c] Lock Acquired to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,154 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [47df259c] Fence failed on vds monique-vds01.tlv.redhat.com which is spm of pool ab2440ae-4286-4465-beed-c2ccc8256f8b - moving pool to non operational >2014-08-19 12:13:30,173 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] The message key SetStoragePoolStatus is missing from bundles/ExecutionMessages >2014-08-19 12:13:30,192 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Lock Acquired to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,194 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Failed to run Fence script on vds:monique-vds05.tlv.redhat.com. >2014-08-19 12:13:30,210 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-08-19 12:13:30,212 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-08-19 12:13:30,213 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-35) [5291d84f] Lock freed to object EngineLock [exclusiveLocks= key: 4290a94f-9e6e-4eed-a517-3401f613ebdf value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,215 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:13:30,238 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Correlation ID: b75ee42, Job ID: 4acbec7c-71aa-4e70-acbf-137ef75744ea, Call Stack: null, Custom Event ID: -1, Message: Fencing failed on Storage Pool Manager monique-vds01.tlv.redhat.com for Data Center DC1. Setting status to Non-Operational. >2014-08-19 12:13:30,290 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:30,291 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Failed to run Fence script on vds:monique-vds01.tlv.redhat.com. >2014-08-19 12:13:30,306 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-08-19 12:13:30,308 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-08-19 12:13:30,309 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Trying to release exclusive lock which does not exist, lock key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affbVDS_FENCE >2014-08-19 12:13:30,310 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-3) [b75ee42] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS_FENCE >, sharedLocks= ] >2014-08-19 12:13:32,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:32,401 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:32,410 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:32,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:32,555 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:32,561 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:37,443 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:37,445 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:37,452 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:37,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:37,598 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:37,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:42,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:42,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:42,496 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:42,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:42,650 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:42,656 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:47,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:47,534 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:47,541 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:47,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:47,690 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:47,695 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:52,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:52,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:52,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:52,728 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:52,730 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:52,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:57,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:57,621 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:57,642 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:13:57,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:13:57,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:13:57,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:02,676 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:02,678 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:02,684 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:02,811 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:02,813 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:02,819 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:07,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:07,718 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:07,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:07,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:07,858 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:07,864 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:12,759 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:12,760 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:12,766 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:12,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:12,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:12,928 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:17,796 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:17,798 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:17,805 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:17,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:17,965 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:17,972 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:22,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:22,862 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:22,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:23,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:23,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:23,056 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:27,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:27,902 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:27,908 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:28,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:28,090 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:28,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:32,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:32,943 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:32,949 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:33,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:33,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:33,140 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:37,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:37,982 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:37,988 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:38,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:38,174 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:38,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:43,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:43,027 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:43,034 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:43,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:43,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:43,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:48,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:48,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:48,083 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:48,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:48,310 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:48,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:53,122 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:53,124 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:53,139 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:53,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:53,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:53,370 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:58,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:58,173 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:58,183 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:14:58,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:14:58,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:14:58,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:03,215 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:03,217 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:03,224 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:03,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:03,458 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:03,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:08,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:08,260 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:08,266 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:08,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:08,498 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:08,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:13,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:13,304 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:13,311 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:13,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:13,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:13,546 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:18,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:18,344 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:18,350 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:18,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:18,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:18,589 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:21,127 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] IrsBroker::Failed::GetStoragePoolInfoVDS due to: TimeoutException: >2014-08-19 12:15:23,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [45b4b475] Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [45b4b475] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-19 12:15:23,273 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [3f2499db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-19 12:15:23,286 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3f2499db] Correlation ID: 3f2499db, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-19 12:15:23,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:23,318 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:23,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:23,323 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:23,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,394 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:23,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:23,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:23,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:23,633 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:28,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:28,468 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:28,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:28,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:28,681 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:28,687 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:33,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:33,433 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:33,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:33,443 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:33,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:33,558 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:33,564 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:33,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:33,725 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:33,733 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:38,598 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:38,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:38,611 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:38,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:38,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:38,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:43,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:43,534 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:43,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:43,538 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:43,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:43,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:43,662 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:43,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:43,809 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:43,824 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:48,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:48,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:48,708 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:48,855 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:48,857 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:48,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:53,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:15:53,629 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:15:53,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:15:53,633 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) spm vds is non responsive, stopping spm selection. >2014-08-19 12:15:53,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:53,751 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:53,758 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:53,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:53,907 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:53,913 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:58,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:58,794 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:58,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:15:58,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:15:58,956 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:15:58,962 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:03,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:03,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:03,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:03,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:03,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:03,837 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:03,844 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:03,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:03,993 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:04,000 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:08,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:08,880 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:08,886 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:09,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:09,033 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:09,039 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:13,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:13,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:13,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:13,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:13,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:13,919 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:13,927 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:14,072 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) [434e9f98] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:14,073 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [434e9f98] Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:14,080 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [434e9f98] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:19,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:19,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:19,035 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:19,112 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:19,114 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:19,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:23,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:23,926 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:23,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:23,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:24,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:24,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:24,079 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:24,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:24,158 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:24,193 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:29,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:29,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:29,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:29,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:29,225 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:29,231 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:34,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:34,031 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:34,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:34,035 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:34,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:34,154 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:34,160 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:34,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:34,262 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:34,321 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:39,191 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:39,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:39,201 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:39,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:39,355 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:39,363 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:44,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:44,132 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:44,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:44,136 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:44,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:44,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:44,246 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:44,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:44,402 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:44,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:49,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:49,289 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:49,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5ecfd6ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:49,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:49,441 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:49,448 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:54,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:16:54,240 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:16:54,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:16:54,244 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) spm vds is non responsive, stopping spm selection. >2014-08-19 12:16:54,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:54,453 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:54,459 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:54,560 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:54,562 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:54,568 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:59,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:59,532 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:59,539 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:16:59,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:16:59,607 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:16:59,656 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:04,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:04,342 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:04,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:04,346 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:04,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:04,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:04,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:04,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:04,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:04,707 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:09,640 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:09,642 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:09,649 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:09,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:09,741 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:09,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:14,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:14,439 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:14,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:14,444 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:14,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:14,691 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:14,698 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:14,779 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:14,780 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:14,792 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:19,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:19,732 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:19,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:19,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:19,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:19,835 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:24,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:24,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:24,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:24,585 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:24,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:24,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:24,790 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:24,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:24,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:24,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:29,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:29,834 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:29,842 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:29,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:29,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:29,967 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:34,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:34,676 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:34,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:34,680 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:34,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:34,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:34,885 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:35,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:35,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:35,014 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:39,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:39,930 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:39,936 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:40,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:40,053 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:40,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:44,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:44,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:44,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:44,791 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:44,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:44,977 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:44,983 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:45,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:45,095 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:45,101 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:50,013 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:50,015 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:50,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:50,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:50,143 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:50,156 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:54,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:17:54,884 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:17:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:17:54,888 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) spm vds is non responsive, stopping spm selection. >2014-08-19 12:17:55,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:55,069 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:55,075 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:17:55,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:17:55,192 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:17:55,209 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:00,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:00,107 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:00,113 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:00,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:00,265 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:00,270 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:05,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:18:05,016 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:18:05,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:18:05,019 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3f2499db] spm vds is non responsive, stopping spm selection. >2014-08-19 12:18:05,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:05,146 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:05,153 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:05,305 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:05,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:05,330 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:10,191 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:10,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:10,200 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:10,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:10,381 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:10,388 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:15,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-19 12:18:15,113 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-19 12:18:15,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-19 12:18:15,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) spm vds is non responsive, stopping spm selection. >2014-08-19 12:18:15,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:15,240 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:15,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:15,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:15,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:15,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-19 12:18:20,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-19 12:18:20,298 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-19 12:18:20,358 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] >2014-08-21 10:48:00,341 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-21 10:48:00,354 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf". >2014-08-21 10:48:00,354 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-21 10:48:00,355 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-21 10:48:00,356 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-21 10:48:00,357 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-21 10:48:00,358 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-21 10:48:00,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-21 10:48:00,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-21 10:48:00,360 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-21 10:48:00,361 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-21 10:48:00,361 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-21 10:48:00,362 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-21 10:48:00,363 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-21 10:48:00,376 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-21 10:48:00,377 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-21 10:48:00,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-21 10:48:00,378 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-21 10:48:00,379 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-21 10:48:00,380 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-21 10:48:00,380 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-21 10:48:00,381 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-21 10:48:00,382 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine". >2014-08-21 10:48:00,382 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-21 10:48:00,383 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-21 10:48:00,384 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-21 10:48:00,384 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-21 10:48:00,385 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "violet-rhel.scl.lab.tlv.redhat.com". >2014-08-21 10:48:00,386 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-21 10:48:00,387 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-21 10:48:00,387 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-21 10:48:00,400 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-21 10:48:00,401 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-21 10:48:00,401 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-21 10:48:00,402 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-21 10:48:00,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-21 10:48:00,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-21 10:48:00,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-21 10:48:00,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-21 10:48:00,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-21 10:48:00,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-21 10:48:00,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-21 10:48:00,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-21 10:48:00,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-21 10:48:00,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-21 10:48:00,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-21 10:48:00,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-21 10:48:00,428 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-21 10:48:00,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-21 10:48:00,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-21 10:48:00,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-21 10:48:00,431 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-21 10:48:00,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-21 10:48:00,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-21 10:48:00,433 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-21 10:48:00,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-21 10:48:00,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-21 10:48:00,435 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-21 10:48:00,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-21 10:48:00,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-21 10:48:00,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-21 10:48:00,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt". >2014-08-21 10:48:00,438 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-21 10:48:00,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-21 10:48:00,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-21 10:48:00,458 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-21 10:48:00,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-21 10:48:00,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-21 10:48:02,760 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-21 10:48:04,336 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-21 10:48:04,348 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/10-sysprep.properties >2014-08-21 10:48:04,801 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-21 10:48:04,801 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-21 10:48:04,806 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-21 10:48:04,807 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-21 10:48:04,808 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-21 10:48:04,884 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-21 10:48:04,885 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-21 10:48:04,898 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-21 10:48:04,900 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-21 10:48:04,903 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-21 10:48:04,932 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-21 10:48:04,980 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-21 10:48:04,982 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-21 10:48:05,005 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-21 10:48:05,235 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-21 10:48:05,320 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-21 10:48:05,365 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-21 10:48:05,367 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-21 10:48:05,368 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-21 10:48:05,369 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-21 10:48:05,370 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-21 10:48:05,505 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-21 10:48:05,506 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-21 10:48:05,507 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-21 10:48:05,509 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-21 10:48:05,512 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-21 10:48:05,514 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-21 10:48:05,765 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-21 10:48:05,766 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-21 10:48:05,767 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-21 10:48:05,769 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-21 10:48:06,107 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-21 10:48:06,119 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-21 10:48:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-21 10:48:06,209 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-21 10:48:06,455 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:06,524 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-08-21 10:48:06,826 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-21 10:48:07,063 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb was added to the Resource Manager >2014-08-21 10:48:07,064 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:07,082 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-08-21 10:48:07,111 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS a8fdd580-c752-4537-9cae-e49e468db7da was added to the Resource Manager >2014-08-21 10:48:07,111 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-21 10:48:07,150 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-08-21 10:48:07,181 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 4290a94f-9e6e-4eed-a517-3401f613ebdf was added to the Resource Manager >2014-08-21 10:48:07,241 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-21 10:48:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-21 10:48:07,249 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-21 10:48:07,254 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-21 10:48:07,255 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-21 10:48:07,256 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-21 10:48:07,268 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-21 10:48:07,269 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-21 10:48:07,293 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-21 10:48:07,275 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-21 10:48:07,343 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-08-21 10:48:07,420 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-21 10:48:07,420 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-21 10:48:07,421 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-21 10:48:07,423 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-21 10:48:11,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:11,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:11,135 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:11,137 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:11,257 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-21 10:48:11,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: UNASSIGNED not exist in string table >2014-08-21 10:48:11,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-21 10:48:11,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-21 10:48:11,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-21 10:48:11,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-21 10:48:11,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-21 10:48:11,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-21 10:48:11,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-21 10:48:11,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-21 10:48:11,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-21 10:48:11,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STOPPED not exist in string table >2014-08-21 10:48:11,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STARTED not exist in string table >2014-08-21 10:48:11,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_ERROR not exist in string table >2014-08-21 10:48:11,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-21 10:48:11,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-21 10:48:11,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-21 10:48:11,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-21 10:48:11,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:11,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:11,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:11,733 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:12,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 2ff5f506 >2014-08-21 10:48:12,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 2ff5f506 >2014-08-21 10:48:12,793 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-08-21 10:48:13,170 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 10:48:14,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:14,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:14,700 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:14,722 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 4290a94f-9e6e-4eed-a517-3401f613ebdf : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:14,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:14,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:14,769 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:14,834 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 10:48:17,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:17,749 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:17,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:17,966 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:18,035 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Server failed to respond, vds_id = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds_name = monique-vds05.tlv.redhat.com, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.ConnectException: Connection refused >2014-08-21 10:48:18,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:18,150 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Server failed to respond, vds_id = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds_name = monique-vds01.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.ConnectException: Connection refused >2014-08-21 10:48:18,163 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:18,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:18,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 26e95ea3 >2014-08-21 10:48:18,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) SpmStopVDSCommand:: vds monique-vds01.tlv.redhat.com is in Connecting status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 10:48:18,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, SpmStopVDSCommand, log id: 26e95ea3 >2014-08-21 10:48:18,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) spm stop on spm failed, stopping spm selection! >2014-08-21 10:48:18,392 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com is non responsive. >2014-08-21 10:48:18,415 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is non responsive. >2014-08-21 10:48:18,496 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-22) ResourceManager::vdsNotResponding entered for Host 4290a94f-9e6e-4eed-a517-3401f613ebdf, monique-vds05.tlv.redhat.com >2014-08-21 10:48:18,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:18,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:18,544 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-23) ResourceManager::vdsNotResponding entered for Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, monique-vds01.tlv.redhat.com >2014-08-21 10:48:18,773 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:18,778 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Opening SSH Soft Fencing session on host monique-vds05.tlv.redhat.com >2014-08-21 10:48:18,966 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:48:18,983 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Opening SSH Soft Fencing session on host monique-vds01.tlv.redhat.com >2014-08-21 10:48:20,231 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-23) [31e5ebe6] Executing SSH Soft Fencing command on host monique-vds01.tlv.redhat.com >2014-08-21 10:48:20,241 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-22) [5c2c8f03] Executing SSH Soft Fencing command on host monique-vds05.tlv.redhat.com >2014-08-21 10:48:21,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:21,756 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:21,774 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 10:48:21,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:21,797 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:21,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:26,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:26,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:26,950 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:26,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:26,973 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:27,025 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:28,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:28,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:28,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:28,450 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) spm vds is non responsive, stopping spm selection. >2014-08-21 10:48:30,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 27f4f4ce >2014-08-21 10:48:30,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GetHardwareInfoVDSCommand, log id: 27f4f4ce >2014-08-21 10:48:31,079 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:31,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf, vds=Host[monique-vds05.tlv.redhat.com,4290a94f-9e6e-4eed-a517-3401f613ebdf]), log id: 42d0cb88 >2014-08-21 10:48:31,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [e1c3f2] FINISH, GetHardwareInfoVDSCommand, log id: 42d0cb88 >2014-08-21 10:48:31,299 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5d41097a-d767-4fa2-baf4-eedd5526f920 Type: StoragePool >2014-08-21 10:48:31,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = 4290a94f-9e6e-4eed-a517-3401f613ebdf), log id: 467de36d >2014-08-21 10:48:31,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [20599bed] FINISH, SetMOMPolicyParametersVDSCommand, log id: 467de36d >2014-08-21 10:48:31,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [20599bed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-08-21 10:48:31,542 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [20599bed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-08-21 10:48:31,583 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20599bed] Initializing Host: monique-vds05.tlv.redhat.com >2014-08-21 10:48:31,591 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [6f5fe43e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4290a94f-9e6e-4eed-a517-3401f613ebdf Type: VDS >2014-08-21 10:48:32,096 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 10:48:32,097 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 10:48:32,107 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 10:48:35,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7acd9284 >2014-08-21 10:48:35,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = monique-vds01.tlv.redhat.com >2014-08-21 10:48:35,237 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-21 10:48:35,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is initializing. Message: Recovering from crash or Initializing >2014-08-21 10:48:35,668 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2014-08-21 10:48:38,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 10:48:38,572 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 10:48:38,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 10:48:38,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-90) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 6ef6fe22 >2014-08-21 10:48:38,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-08-21 10:48:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) HostName = monique-vds01.tlv.redhat.com >2014-08-21 10:48:38,668 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-08-21 10:48:38,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, SpmStopVDSCommand, log id: 6ef6fe22 >2014-08-21 10:48:38,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) spm stop succeeded, continuing with spm selection >2014-08-21 10:48:38,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId 1, LVER 5 >2014-08-21 10:48:38,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=1, prevLVER=5, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 3b72654e >2014-08-21 10:48:38,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling started: taskId = 0adefff1-d9e8-467a-8aab-02828a379473 >2014-08-21 10:48:39,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 4a0a0394 >2014-08-21 10:48:39,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 4a0a0394 >2014-08-21 10:48:39,712 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:48:39,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling ended: taskId = 0adefff1-d9e8-467a-8aab-02828a379473 task status = finished >2014-08-21 10:48:39,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: caf5747 >2014-08-21 10:48:39,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [607b2f00] FINISH, GetHardwareInfoVDSCommand, log id: caf5747 >2014-08-21 10:48:39,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) spmStart polling ended, spm status: SPM >2014-08-21 10:48:39,902 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [607b2f00] Cancelling the recovery from crash timer for VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb because vds started initializing >2014-08-21 10:48:39,905 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [395fc907] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 10:48:39,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-90) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=0adefff1-d9e8-467a-8aab-02828a379473), log id: 422b00b1 >2014-08-21 10:48:39,958 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [23375153] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 10:48:39,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, HSMClearTaskVDSCommand, log id: 422b00b1 >2014-08-21 10:48:39,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5907a8b6, log id: 3b72654e >2014-08-21 10:48:39,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 10:48:40,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63a68eed >2014-08-21 10:48:40,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 10:48:40,125 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 10:48:40,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 7296b586 >2014-08-21 10:48:40,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 10:48:40,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 46a1d6fe >2014-08-21 10:48:40,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46a1d6fe >2014-08-21 10:48:40,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7296b586 >2014-08-21 10:48:40,351 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool DC1 >2014-08-21 10:48:40,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 63a68eed >2014-08-21 10:48:40,920 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [23375153] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 10:48:40,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 3d38bcff >2014-08-21 10:48:41,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, ConnectStoragePoolVDSCommand, log id: 3d38bcff >2014-08-21 10:48:41,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 396ae4bf >2014-08-21 10:48:41,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [23375153] FINISH, SetMOMPolicyParametersVDSCommand, log id: 396ae4bf >2014-08-21 10:48:41,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [23375153] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-08-21 10:48:41,758 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [23375153] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 10:48:41,782 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [23375153] Initializing Host: monique-vds01.tlv.redhat.com >2014-08-21 10:48:41,787 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [43495829] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 10:49:00,972 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginAdminUserCommand internal: false. >2014-08-21 10:49:01,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 10:49:10,799 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-21 10:49:10,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 10:49:31,596 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp--127.0.0.1-8702-3) [50a32d2e] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-21 10:49:35,104 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp--127.0.0.1-8702-5) [541b55be] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-21 10:50:02,674 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:02,675 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:02,677 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [1d2e10c0] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 10:50:03,024 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:03,025 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 10:50:03,271 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 10:50:03,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 1d2e10c0, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:28,027 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-21 11:01:28,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 11:01:28,080 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [1d7ce352] Running command: LogoutBySessionCommand internal: false. >2014-08-21 11:01:28,095 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [e54cf86] Running command: LogoutUserCommand internal: false. >2014-08-21 11:01:28,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [e54cf86] Correlation ID: e54cf86, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-21 11:01:38,278 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-21 11:01:38,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-21 11:01:41,075 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:41,088 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:41,089 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_a5eb1b3a-12f4-409c] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT_CREDENTIALS >2014-08-21 11:01:41,125 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but Agent credentials are missing.] >2014-08-21 11:01:43,313 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,314 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,315 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_98696063-2f44-48ae] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:43,386 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,387 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:43,520 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:43,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_98696063-2f44-48ae, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:45,375 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,376 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,388 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_7f4f56e2-55b9-4b13] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:45,409 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,410 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:45,464 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:45,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_7f4f56e2-55b9-4b13, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:46,177 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,178 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,205 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_2a1978a8-6b0f-4127] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT_CREDENTIALS >2014-08-21 11:01:46,207 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but Agent credentials are missing.] >2014-08-21 11:01:46,956 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,957 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,958 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_ec8262e7-74ad-4005] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:46,989 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:46,990 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,037 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:47,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_ec8262e7-74ad-4005, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:47,328 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,329 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,330 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_3b3c7bc9-77e9-40bd] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:47,367 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:47,415 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:47,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_3b3c7bc9-77e9-40bd, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:48,077 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,077 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,078 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_e39f10a6-efbd-484d] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_PM_ENABLED_WITHOUT_AGENT >2014-08-21 11:01:48,080 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Power Management is enabled for Host but no Agent type selected.] >2014-08-21 11:01:48,753 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,753 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,754 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_154ae1e2-f951-4d60] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:48,801 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,802 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:48,838 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:48,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_154ae1e2-f951-4d60, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:49,134 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,135 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,136 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_bf231ac4-804d-4af0] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:49,172 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,173 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,219 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:49,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_bf231ac4-804d-4af0, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:49,844 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,845 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:49,846 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_39680d7a-2801-4adc] CanDoAction of action UpdateVds failed. Reasons:VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_AGENT_NOT_SUPPORTED >2014-08-21 11:01:49,847 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Cannot edit Host. Selected Power Management Agent is not supported.] >2014-08-21 11:01:50,288 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,289 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,290 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_e390248d-1a87-4d40] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:50,296 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,297 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,324 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:50,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_e390248d-1a87-4d40, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:50,514 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,515 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,516 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_2c61516a-78ec-4eb8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:50,524 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,525 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:50,553 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:50,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_2c61516a-78ec-4eb8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:01:51,101 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,102 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,103 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_1bc318ee-0f24-4841] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:01:51,172 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,173 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:01:51,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:01:51,574 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:01:51,613 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:01:51,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 29004677 >2014-08-21 11:01:52,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 29004677 >2014-08-21 11:01:52,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_1bc318ee-0f24-4841, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:02:00,736 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:00,977 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:02:01,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Correlation ID: vms_create_5f3cb986-173e-480b, Job ID: 2586dd42-6099-4f5b-b107-28c0b581d609, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was created by admin. >2014-08-21 11:02:01,685 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-7) [vms_create_5f3cb986-173e-480b] Lock freed to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:08,550 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-4) [nics_create_89fea944-58d1-4cc6] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:02:08,653 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-4) [d2c325f] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:02:08,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [d2c325f] Correlation ID: d2c325f, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_ha. (User: admin) >2014-08-21 11:02:08,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [d2c325f] Correlation ID: nics_create_89fea944-58d1-4cc6, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_ha. (User: admin) >2014-08-21 11:02:12,419 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM_DISK_BOOT >, sharedLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >] >2014-08-21 11:02:12,488 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Running command: AddDiskCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:12,489 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-1) [disks_create_4bca8449-2ee1-4ca6] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_ha_Disk1 >2014-08-21 11:02:12,876 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:13,029 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM_DISK_BOOT >, sharedLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >] >2014-08-21 11:02:13,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = 46aa71cf-39b1-4d85-a28b-41cddf3c9315, newImageDescription = ), log id: e1a9f15 >2014-08-21 11:02:13,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:02:13,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [2e26c3b4] FINISH, CreateImageVDSCommand, return: 46aa71cf-39b1-4d85-a28b-41cddf3c9315, log id: e1a9f15 >2014-08-21 11:02:13,476 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [2e26c3b4] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 3c7421d2-0b83-458a-b920-5417c9499dfd >2014-08-21 11:02:13,477 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [2e26c3b4] CommandMultiAsyncTasks::AttachTask: Attaching task 07c43944-f88f-4e80-abfd-ce959107ca88 to command 3c7421d2-0b83-458a-b920-5417c9499dfd. >2014-08-21 11:02:13,526 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [2e26c3b4] Adding task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:02:13,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [2e26c3b4] Correlation ID: disks_create_4bca8449-2ee1-4ca6, Job ID: 28d89b8a-1e66-4b03-8dde-b5f31e90cd9b, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_ha_Disk1 was initiated on VM vm_ha by admin. >2014-08-21 11:02:13,610 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [2e26c3b4] BaseAsyncTask::startPollingTask: Starting to poll task 07c43944-f88f-4e80-abfd-ce959107ca88. >2014-08-21 11:02:16,265 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-95) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-08-21 11:02:16,287 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-95) SPMAsyncTask::PollTask: Polling task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:02:16,296 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-95) BaseAsyncTask::OnTaskEndSuccess: Task 07c43944-f88f-4e80-abfd-ce959107ca88 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:02:16,297 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-95) CommandAsyncTask::EndActionIfNecessary: All tasks of command 3c7421d2-0b83-458a-b920-5417c9499dfd has ended -> executing endAction >2014-08-21 11:02:16,298 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-95) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 3c7421d2-0b83-458a-b920-5417c9499dfd): calling endAction . >2014-08-21 11:02:16,300 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:02:16,318 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-45) [disks_create_4bca8449-2ee1-4ca6] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:02:16,338 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:02:16,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, imageId = 46aa71cf-39b1-4d85-a28b-41cddf3c9315), log id: 11616204 >2014-08-21 11:02:16,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-45) [2e26c3b4] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@f03cb394, log id: 11616204 >2014-08-21 11:02:16,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) Correlation ID: disks_create_4bca8449-2ee1-4ca6, Call Stack: null, Custom Event ID: -1, Message: The disk vm_ha_Disk1 was successfully added to VM vm_ha. >2014-08-21 11:02:16,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:02:16,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:02:16,410 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-45) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 07c43944-f88f-4e80-abfd-ce959107ca88 >2014-08-21 11:02:16,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 07c43944-f88f-4e80-abfd-ce959107ca88), log id: 7d75be37 >2014-08-21 11:02:16,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=07c43944-f88f-4e80-abfd-ce959107ca88), log id: 3d1b4d3a >2014-08-21 11:02:16,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, HSMClearTaskVDSCommand, log id: 3d1b4d3a >2014-08-21 11:02:16,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, SPMClearTaskVDSCommand, log id: 7d75be37 >2014-08-21 11:02:16,489 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-45) BaseAsyncTask::RemoveTaskFromDB: Removed task 07c43944-f88f-4e80-abfd-ce959107ca88 from DataBase >2014-08-21 11:02:16,490 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-45) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 3c7421d2-0b83-458a-b920-5417c9499dfd >2014-08-21 11:02:25,488 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:02:25,773 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, IsVmDuringInitiatingVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75), log id: 6e2e2e52 >2014-08-21 11:02:25,774 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6e2e2e52 >2014-08-21 11:02:26,003 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Running command: RunVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:02:26,352 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@482bb400), log id: 6e9a15b2 >2014-08-21 11:02:26,535 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6e9a15b2 >2014-08-21 11:02:26,599 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 1cdba630 >2014-08-21 11:02:26,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 2fbf0907 >2014-08-21 11:02:27,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiNmFkMWZkMzItYTE0NC00MmYw >LThmYWYtNzk4YmQ2NDM4YjVhIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=2fa59eb2-1abe-4dba-a2d6-3a1534d648d3}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=063dada5-e1f8-4d26-a1a6-3f36e9dbaeaa}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8a8b77f4-9743-4fde-a10a-e964561baeaf}, {shared=false, volumeID=46aa71cf-39b1-4d85-a28b-41cddf3c9315, index=0, iface=virtio, imageID=0045c380-09dd-4340-ac3a-b22edee60ca8, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=0045c380-09dd-4340-ac3a-b22edee60ca8}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=45954a54-2239-49be-abec-0db12b68a8ae}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=54a9ed8e-e144-40f4-a409-21ad16885778}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=8a1d6f2c-cfae-4c6a-b1ab-2f288c07e9b8}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:02:27,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2fbf0907 >2014-08-21 11:02:27,438 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1cdba630 >2014-08-21 11:02:27,438 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:02:27,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_9dc5ee87-4dc6-474d] Correlation ID: vms_syncAction_9dc5ee87-4dc6-474d, Job ID: c13c1f72-b942-4383-9ab7-654de767ef9c, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:02:29,634 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:02:29,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-42) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[64ff4f66-e697-4a93-b7ff-dba82eb02b75]), log id: 260020d >2014-08-21 11:02:29,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-42) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=8055, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@6d26a4b2, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 260020d >2014-08-21 11:02:29,683 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Received a spice Device without an address when processing VM 64ff4f66-e697-4a93-b7ff-dba82eb02b75 devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:02:40,116 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:40,342 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:02:40,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Correlation ID: vms_create_84d3bbe4-566e-4a0e, Job ID: 0296d326-ab1b-401a-8cc0-fd241f41fbac, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was created by admin. >2014-08-21 11:02:40,551 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-2) [vms_create_84d3bbe4-566e-4a0e] Lock freed to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:02:41,842 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-6) [nics_create_07c7acc3-03d2-4341] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:02:41,871 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-6) [5f257142] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:02:41,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f257142] Correlation ID: 5f257142, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_2. (User: admin) >2014-08-21 11:02:41,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f257142] Correlation ID: nics_create_07c7acc3-03d2-4341, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_2. (User: admin) >2014-08-21 11:02:42,813 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM_DISK_BOOT >, sharedLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >] >2014-08-21 11:02:42,843 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Running command: AddDiskCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:42,845 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-1) [disks_create_a6425d58-a616-44b8] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_2_Disk1 >2014-08-21 11:02:42,878 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [492dc420] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:02:42,887 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-1) [492dc420] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM_DISK_BOOT >, sharedLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >] >2014-08-21 11:02:42,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = e44ee009-3a78-40e2-b024-e236b205a702, newImageDescription = ), log id: 7bcebfdc >2014-08-21 11:02:42,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:02:42,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-1) [492dc420] FINISH, CreateImageVDSCommand, return: e44ee009-3a78-40e2-b024-e236b205a702, log id: 7bcebfdc >2014-08-21 11:02:42,975 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [492dc420] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 58878f92-8d69-4c7b-8275-678acfd01cfc >2014-08-21 11:02:42,976 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [492dc420] CommandMultiAsyncTasks::AttachTask: Attaching task 6dba1c95-354d-4318-bb82-d1658b766252 to command 58878f92-8d69-4c7b-8275-678acfd01cfc. >2014-08-21 11:02:43,017 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [492dc420] Adding task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:02:43,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [492dc420] Correlation ID: disks_create_a6425d58-a616-44b8, Job ID: ac69d713-b229-4047-b594-b2876b141799, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_2_Disk1 was initiated on VM vm_2 by admin. >2014-08-21 11:02:43,084 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [492dc420] BaseAsyncTask::startPollingTask: Starting to poll task 6dba1c95-354d-4318-bb82-d1658b766252. >2014-08-21 11:02:46,305 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-93) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now >2014-08-21 11:02:46,310 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) SPMAsyncTask::PollTask: Polling task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:02:46,328 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) BaseAsyncTask::OnTaskEndSuccess: Task 6dba1c95-354d-4318-bb82-d1658b766252 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:02:46,329 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::EndActionIfNecessary: All tasks of command 58878f92-8d69-4c7b-8275-678acfd01cfc has ended -> executing endAction >2014-08-21 11:02:46,330 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 58878f92-8d69-4c7b-8275-678acfd01cfc): calling endAction . >2014-08-21 11:02:46,331 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:02:46,347 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-33) [disks_create_a6425d58-a616-44b8] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:02:46,372 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:02:46,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, imageId = e44ee009-3a78-40e2-b024-e236b205a702), log id: 52c24bbc >2014-08-21 11:02:46,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [492dc420] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8c462649, log id: 52c24bbc >2014-08-21 11:02:46,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) Correlation ID: disks_create_a6425d58-a616-44b8, Call Stack: null, Custom Event ID: -1, Message: The disk vm_2_Disk1 was successfully added to VM vm_2. >2014-08-21 11:02:46,420 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:02:46,421 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:02:46,424 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 6dba1c95-354d-4318-bb82-d1658b766252 >2014-08-21 11:02:46,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 6dba1c95-354d-4318-bb82-d1658b766252), log id: 26dcf806 >2014-08-21 11:02:46,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=6dba1c95-354d-4318-bb82-d1658b766252), log id: 605783fa >2014-08-21 11:02:46,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMClearTaskVDSCommand, log id: 605783fa >2014-08-21 11:02:46,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMClearTaskVDSCommand, log id: 26dcf806 >2014-08-21 11:02:46,502 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) BaseAsyncTask::RemoveTaskFromDB: Removed task 6dba1c95-354d-4318-bb82-d1658b766252 from DataBase >2014-08-21 11:02:46,503 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 58878f92-8d69-4c7b-8275-678acfd01cfc >2014-08-21 11:02:53,604 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:02:53,645 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, IsVmDuringInitiatingVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330), log id: 6fd9c9f5 >2014-08-21 11:02:53,656 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6fd9c9f5 >2014-08-21 11:02:53,779 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Running command: RunVmCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:02:53,992 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@ddc8f42b), log id: 696ffcaf >2014-08-21 11:02:54,026 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, UpdateVmDynamicDataVDSCommand, log id: 696ffcaf >2014-08-21 11:02:54,095 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, vm=VM [vm_2]), log id: 2a5556b0 >2014-08-21 11:02:54,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, vm=VM [vm_2]), log id: 2cab4844 >2014-08-21 11:02:54,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiZmE5YmU3OTMtNzFhMC00MmI1 >LThiZGItMzBhM2IwMDhhZjE1IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=51d5d3f5-e402-4242-81a6-f24bb8c563c8}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=1c30cb18-db5c-473f-9f6c-64aaaf2f4983}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=0e4b0a84-54d5-4045-92ad-a858d81fad19}, {shared=false, volumeID=e44ee009-3a78-40e2-b024-e236b205a702, index=0, iface=virtio, imageID=3b7c6a02-e649-4b99-91a0-62e15e88e85e, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=3b7c6a02-e649-4b99-91a0-62e15e88e85e}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=316c1233-4d21-48be-885d-a0bafb231cc9}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=4bdd9ddc-8f75-4130-b070-bca70ecee19b}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=5ef82c7e-442b-40d0-89bb-a0c66e778d62}],vmName=vm_2,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:02:54,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2cab4844 >2014-08-21 11:02:54,296 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2a5556b0 >2014-08-21 11:02:54,297 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:02:54,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [vms_syncAction_3f06bbf1-f98f-4299] Correlation ID: vms_syncAction_3f06bbf1-f98f-4299, Job ID: 3fa2d29f-4b21-4dbb-8d5f-c340f93c54fd, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:02:57,942 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) VM vm_2 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:02:57,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-29) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[8e7744c0-9b41-4db0-a62c-d85d7ddaa330]), log id: 3aabcd03 >2014-08-21 11:02:57,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-29) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=8e7744c0-9b41-4db0-a62c-d85d7ddaa330, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=8237, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@65209e0c, vmName=vm_2, fileTransferEnable=true, cpuType=Conroe}], log id: 3aabcd03 >2014-08-21 11:02:57,973 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Received a spice Device without an address when processing VM 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5903, type=graphics} >2014-08-21 11:03:07,567 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_dc5f5594-bf04-4869] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:03:07,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:07,791 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:07,801 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:07,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:07,920 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:07,963 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:07,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3fdbcf45 >2014-08-21 11:03:08,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3fdbcf45 >2014-08-21 11:03:08,239 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:08,283 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 65ed7a8 >2014-08-21 11:03:08,284 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 11:03:08,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 20b810dc >2014-08-21 11:03:08,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 518ac31f >2014-08-21 11:03:08,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:03:08,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SpmStopVDSCommand, log id: 518ac31f >2014-08-21 11:03:08,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, ResetIrsVDSCommand, log id: 20b810dc >2014-08-21 11:03:08,399 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SetVdsStatusVDSCommand, log id: 65ed7a8 >2014-08-21 11:03:08,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:08,516 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:08,549 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:08,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7d8959ef >2014-08-21 11:03:08,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, FenceVdsVDSCommand, return: Done, log id: 7d8959ef >2014-08-21 11:03:08,680 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:03:13,682 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:03:13,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:13,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:13,796 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [270799bf] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:13,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3af20533 >2014-08-21 11:03:13,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 3af20533 >2014-08-21 11:03:13,927 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [270799bf] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:03:13,930 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] START, SetVmStatusVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75, status = Down, exit status = Error), log id: 7a4d95ee >2014-08-21 11:03:13,947 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [270799bf] FINISH, SetVmStatusVDSCommand, log id: 7a4d95ee >2014-08-21 11:03:13,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [270799bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_ha was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:03:13,997 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:03:14,030 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] START, SetVmStatusVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330, status = Down, exit status = Error), log id: 3804f5ec >2014-08-21 11:03:14,038 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [7ea335eb] FINISH, SetVmStatusVDSCommand, log id: 3804f5ec >2014-08-21 11:03:14,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [7ea335eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_2 was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:03:14,102 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:03:14,177 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 5dc89493 >2014-08-21 11:03:14,196 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (ajp--127.0.0.1-8702-6) [30a8f4dc] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 5dc89493 >2014-08-21 11:03:14,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [30a8f4dc] Correlation ID: 270799bf, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:03:14,304 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [6d728b03] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:14,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:14,592 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:14,597 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:03:14,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5cec8ccf >2014-08-21 11:03:14,676 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 5cec8ccf >2014-08-21 11:03:14,748 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, IsVmDuringInitiatingVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75), log id: 19d16c91 >2014-08-21 11:03:14,750 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 19d16c91 >2014-08-21 11:03:14,852 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] Running command: RunVmCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:03:14,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:14,925 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:14,923 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, UpdateVmDynamicDataVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@25271673), log id: 7554c86e >2014-08-21 11:03:14,972 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:14,974 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7554c86e >2014-08-21 11:03:14,987 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, CreateVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 2cf6f5d1 >2014-08-21 11:03:14,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5deadebd >2014-08-21 11:03:14,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, CreateVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, vm=VM [vm_ha]), log id: 6ca863f >2014-08-21 11:03:15,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=063dada5-e1f8-4d26-a1a6-3f36e9dbaeaa}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8a8b77f4-9743-4fde-a10a-e964561baeaf}, {shared=false, index=0, volumeID=46aa71cf-39b1-4d85-a28b-41cddf3c9315, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, imageID=0045c380-09dd-4340-ac3a-b22edee60ca8, specParams={}, optional=false, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, readonly=false, deviceId=0045c380-09dd-4340-ac3a-b22edee60ca8}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=45954a54-2239-49be-abec-0db12b68a8ae}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=54a9ed8e-e144-40f4-a409-21ad16885778}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=8a1d6f2c-cfae-4c6a-b1ab-2f288c07e9b8}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:03:15,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 5deadebd >2014-08-21 11:03:15,200 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:03:15,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, CreateVDSCommand, log id: 6ca863f >2014-08-21 11:03:15,253 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2cf6f5d1 >2014-08-21 11:03:15,254 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:03:15,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1dae0525] Correlation ID: 1dae0525, Job ID: 5df08973-9395-4cd0-b6ff-444eb824ad60, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:03:17,844 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:03:17,850 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) [799af561] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 11:03:17,894 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [799af561] Correlation ID: 799af561, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:03:17,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:03:17,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:03:17,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a02669c >2014-08-21 11:03:18,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling started: taskId = 157eeb64-27cb-4528-a3f9-8bd64a5ccf31 >2014-08-21 11:03:18,381 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from WaitForLaunch --> PoweringUp >2014-08-21 11:03:18,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-9) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb], vmIds=[64ff4f66-e697-4a93-b7ff-dba82eb02b75]), log id: 720b036c >2014-08-21 11:03:18,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-9) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.2.45, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=16360, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@66cf007, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 720b036c >2014-08-21 11:03:18,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Received a spice Device without an address when processing VM 64ff4f66-e697-4a93-b7ff-dba82eb02b75 devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.2.45}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:03:19,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling ended: taskId = 157eeb64-27cb-4528-a3f9-8bd64a5ccf31 task status = finished >2014-08-21 11:03:19,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] spmStart polling ended, spm status: SPM >2014-08-21 11:03:19,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=157eeb64-27cb-4528-a3f9-8bd64a5ccf31), log id: 1b8bf2e9 >2014-08-21 11:03:19,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] FINISH, HSMClearTaskVDSCommand, log id: 1b8bf2e9 >2014-08-21 11:03:19,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29b826cf, log id: 5a02669c >2014-08-21 11:03:19,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) [799af561] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-21 11:03:19,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [799af561] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-21 11:03:19,148 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:03:19,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 1b42a7d9 >2014-08-21 11:03:19,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:03:19,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 4356d50d >2014-08-21 11:03:19,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4356d50d >2014-08-21 11:03:19,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [799af561] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b42a7d9 >2014-08-21 11:03:19,235 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [799af561] Discovered no tasks on Storage Pool DC1 >2014-08-21 11:03:20,201 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:03:20,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:03:20,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:03:20,389 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:03:20,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 64a7bffa >2014-08-21 11:03:20,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 64a7bffa >2014-08-21 11:03:20,522 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:03:20,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 681e3c44, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:03:20,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_dc5f5594-bf04-4869, Job ID: 59fd347d-0f75-45cf-9284-be26431c47ad, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:03:20,546 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:03:36,200 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Setting new tasks map. The map contains now 1 tasks >2014-08-21 11:04:06,201 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:04:06,202 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-39) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:04:16,334 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) VM vm_ha 64ff4f66-e697-4a93-b7ff-dba82eb02b75 moved from PoweringUp --> Up >2014-08-21 11:04:16,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: 1dae0525, Job ID: 5df08973-9395-4cd0-b6ff-444eb824ad60, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:04:16,347 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-12) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:04:59,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:04:59,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:04:59,420 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:05:03,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 6ade3f33 >2014-08-21 11:05:03,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 6ade3f33 >2014-08-21 11:05:03,623 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [3b9cb023] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:05:03,634 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [2505d1b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:05:03,643 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:05:03,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b5f814d >2014-08-21 11:05:03,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5b5f814d >2014-08-21 11:05:03,910 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:05:03,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 3e53e374 >2014-08-21 11:05:04,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 3e53e374 >2014-08-21 11:05:04,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:05:04,286 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [285aee97] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:05:04,318 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-12) [285aee97] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:05:04,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5b7c22a0 >2014-08-21 11:05:04,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5b7c22a0 >2014-08-21 11:05:04,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 43193aa7 >2014-08-21 11:05:04,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [285aee97] FINISH, SetMOMPolicyParametersVDSCommand, log id: 43193aa7 >2014-08-21 11:05:04,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:05:04,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [285aee97] Correlation ID: 2505d1b2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:05:04,800 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:22,543 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,544 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,545 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_update_e9d8372c-4237-4750] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:22,551 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,552 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:22,608 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:06:22,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_update_e9d8372c-4237-4750, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:06:23,047 INFO [org.ovirt.engine.core.bll.StopVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] Running command: StopVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:23,053 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] START, DestroyVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, force=false, secondsToWait=0, gracefully=false, reason=), log id: 6cf12495 >2014-08-21 11:06:23,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] START, DestroyVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=64ff4f66-e697-4a93-b7ff-dba82eb02b75, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3700dcc4 >2014-08-21 11:06:23,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] FINISH, DestroyVDSCommand, log id: 3700dcc4 >2014-08-21 11:06:23,570 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] FINISH, DestroyVmVDSCommand, return: Down, log id: 6cf12495 >2014-08-21 11:06:23,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_dedb8fb6-f209-4a68] Correlation ID: vms_syncAction_dedb8fb6-f209-4a68, Job ID: 3308492d-de05-409d-8ba9-fcf0f660e780, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha powered off by admin (Host: monique-vds01.tlv.redhat.com) (Reason: Not Specified). >2014-08-21 11:06:23,678 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-29) [f207e07] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:06:23,953 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Lock Acquired to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:06:23,980 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Running command: RemoveVmCommand internal: false. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:23,984 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] START, SetVmStatusVDSCommand( vmId = 64ff4f66-e697-4a93-b7ff-dba82eb02b75, status = ImageLocked, exit status = Normal), log id: 3db87b61 >2014-08-21 11:06:23,999 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] FINISH, SetVmStatusVDSCommand, log id: 3db87b61 >2014-08-21 11:06:24,037 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_8de37fce-0e67-49f2] Lock freed to object EngineLock [exclusiveLocks= key: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 value: VM >, sharedLocks= ] >2014-08-21 11:06:24,074 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-3) [4f0d7561] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 64ff4f66-e697-4a93-b7ff-dba82eb02b75 Type: VM >2014-08-21 11:06:24,097 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:06:24,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 0045c380-09dd-4340-ac3a-b22edee60ca8, postZeros = false, forceDelete = false), log id: 1df21d6e >2014-08-21 11:06:24,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [2bc6f28a] FINISH, DeleteImageGroupVDSCommand, log id: 1df21d6e >2014-08-21 11:06:24,483 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [2bc6f28a] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 1728d2e1-9c68-45d4-93ee-6da4e96ab872 >2014-08-21 11:06:24,484 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [2bc6f28a] CommandMultiAsyncTasks::AttachTask: Attaching task 475de8eb-b6f7-4096-813c-69678db7fa4d to command 1728d2e1-9c68-45d4-93ee-6da4e96ab872. >2014-08-21 11:06:24,583 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [2bc6f28a] Adding task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:24,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [2bc6f28a] Correlation ID: vms_delete_8de37fce-0e67-49f2, Job ID: 0cbadf15-a454-49b2-a94f-4354ef542b53, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was successfully removed. >2014-08-21 11:06:24,775 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [2bc6f28a] BaseAsyncTask::startPollingTask: Starting to poll task 475de8eb-b6f7-4096-813c-69678db7fa4d. >2014-08-21 11:06:25,149 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Lock Acquired to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:06:25,179 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Running command: RemoveVmCommand internal: false. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:06:25,180 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] START, SetVmStatusVDSCommand( vmId = 8e7744c0-9b41-4db0-a62c-d85d7ddaa330, status = ImageLocked, exit status = Normal), log id: 47a40e4f >2014-08-21 11:06:25,183 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] FINISH, SetVmStatusVDSCommand, log id: 47a40e4f >2014-08-21 11:06:25,185 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-1) [vms_delete_87782caf-2d36-4d4b] Lock freed to object EngineLock [exclusiveLocks= key: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 value: VM >, sharedLocks= ] >2014-08-21 11:06:25,201 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-1) [61bce53] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 8e7744c0-9b41-4db0-a62c-d85d7ddaa330 Type: VM >2014-08-21 11:06:25,205 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-1) [3c3e33] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:06:25,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-1) [3c3e33] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 3b7c6a02-e649-4b99-91a0-62e15e88e85e, postZeros = false, forceDelete = false), log id: 7967e579 >2014-08-21 11:06:25,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-1) [3c3e33] FINISH, DeleteImageGroupVDSCommand, log id: 7967e579 >2014-08-21 11:06:25,391 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-1) [3c3e33] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 >2014-08-21 11:06:25,392 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-1) [3c3e33] CommandMultiAsyncTasks::AttachTask: Attaching task e7abe633-747a-4cdd-9ace-e41ecf623c66 to command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71. >2014-08-21 11:06:25,445 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-1) [3c3e33] Adding task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:25,653 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3c3e33] Correlation ID: vms_delete_87782caf-2d36-4d4b, Job ID: f94b3181-424c-45dd-b0f5-9185b659d077, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was successfully removed. >2014-08-21 11:06:25,654 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-1) [3c3e33] BaseAsyncTask::startPollingTask: Starting to poll task e7abe633-747a-4cdd-9ace-e41ecf623c66. >2014-08-21 11:06:26,041 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,042 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,043 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_48264477-b9d4-46d8] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:06:26,108 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,109 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:06:26,347 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-08-21 11:06:26,357 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:06:26,358 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:06:26,360 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:26,394 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:06:26,395 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) BaseAsyncTask::OnTaskEndSuccess: Task 475de8eb-b6f7-4096-813c-69678db7fa4d (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:26,396 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::EndActionIfNecessary: All tasks of command 1728d2e1-9c68-45d4-93ee-6da4e96ab872 has ended -> executing endAction >2014-08-21 11:06:26,397 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 1728d2e1-9c68-45d4-93ee-6da4e96ab872): calling endAction . >2014-08-21 11:06:26,398 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:26,399 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:06:26,426 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) BaseAsyncTask::OnTaskEndSuccess: Task e7abe633-747a-4cdd-9ace-e41ecf623c66 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:26,427 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::EndActionIfNecessary: All tasks of command 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 has ended -> executing endAction >2014-08-21 11:06:26,428 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-92) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 918c6e62-1e21-4c2a-8a4d-beb1a63cad71): calling endAction . >2014-08-21 11:06:26,439 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:06:26,441 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:06:26,444 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:06:26,444 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:06:26,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3fba3289 >2014-08-21 11:06:26,455 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 475de8eb-b6f7-4096-813c-69678db7fa4d >2014-08-21 11:06:26,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 475de8eb-b6f7-4096-813c-69678db7fa4d), log id: 2bf402d0 >2014-08-21 11:06:26,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=475de8eb-b6f7-4096-813c-69678db7fa4d), log id: 901d65c >2014-08-21 11:06:26,496 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:06:26,497 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:06:26,498 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:06:26,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] FINISH, HSMClearTaskVDSCommand, log id: 901d65c >2014-08-21 11:06:26,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] FINISH, SPMClearTaskVDSCommand, log id: 2bf402d0 >2014-08-21 11:06:26,564 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task e7abe633-747a-4cdd-9ace-e41ecf623c66 >2014-08-21 11:06:26,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = e7abe633-747a-4cdd-9ace-e41ecf623c66), log id: 1eb017ca >2014-08-21 11:06:26,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=e7abe633-747a-4cdd-9ace-e41ecf623c66), log id: 3ae9de83 >2014-08-21 11:06:26,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] FINISH, HSMClearTaskVDSCommand, log id: 3ae9de83 >2014-08-21 11:06:26,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] FINISH, SPMClearTaskVDSCommand, log id: 1eb017ca >2014-08-21 11:06:26,632 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] BaseAsyncTask::RemoveTaskFromDB: Removed task 475de8eb-b6f7-4096-813c-69678db7fa4d from DataBase >2014-08-21 11:06:26,633 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-22) [vms_delete_8de37fce-0e67-49f2] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 1728d2e1-9c68-45d4-93ee-6da4e96ab872 >2014-08-21 11:06:26,645 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] BaseAsyncTask::RemoveTaskFromDB: Removed task e7abe633-747a-4cdd-9ace-e41ecf623c66 from DataBase >2014-08-21 11:06:26,645 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-38) [vms_delete_87782caf-2d36-4d4b] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 918c6e62-1e21-4c2a-8a4d-beb1a63cad71 >2014-08-21 11:06:26,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3fba3289 >2014-08-21 11:06:26,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_48264477-b9d4-46d8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:06:27,722 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:27,822 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:06:27,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Correlation ID: vms_create_ff88116a-b971-40b5, Job ID: d080ee01-2a6a-4952-bb10-a963e06fd570, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was created by admin. >2014-08-21 11:06:27,972 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-8) [vms_create_ff88116a-b971-40b5] Lock freed to object EngineLock [exclusiveLocks= key: vm_ha value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:28,937 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-7) [nics_create_bd9ab51d-a322-41b5] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:06:28,990 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-7) [4171ca3d] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:06:29,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4171ca3d] Correlation ID: 4171ca3d, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_ha. (User: admin) >2014-08-21 11:06:29,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4171ca3d] Correlation ID: nics_create_bd9ab51d-a322-41b5, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_ha. (User: admin) >2014-08-21 11:06:29,491 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM_DISK_BOOT >, sharedLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >] >2014-08-21 11:06:29,510 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Running command: AddDiskCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:29,511 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-6) [disks_create_2b55890d-1f35-46db] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_ha_Disk1 >2014-08-21 11:06:29,612 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-6) [2cd61457] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:29,620 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-6) [2cd61457] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM_DISK_BOOT >, sharedLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >] >2014-08-21 11:06:29,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = de7bf293-5790-426c-9f8a-c391b1396c8a, newImageDescription = ), log id: 70929ab6 >2014-08-21 11:06:29,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:06:29,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-6) [2cd61457] FINISH, CreateImageVDSCommand, return: de7bf293-5790-426c-9f8a-c391b1396c8a, log id: 70929ab6 >2014-08-21 11:06:29,768 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-6) [2cd61457] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command c15aff19-5992-40de-9c84-1e24980ea3d7 >2014-08-21 11:06:29,769 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-6) [2cd61457] CommandMultiAsyncTasks::AttachTask: Attaching task ed0aef1f-d35a-45a8-88af-cb265e908409 to command c15aff19-5992-40de-9c84-1e24980ea3d7. >2014-08-21 11:06:29,784 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-6) [2cd61457] Adding task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:29,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [2cd61457] Correlation ID: disks_create_2b55890d-1f35-46db, Job ID: c9ceedf9-2878-4cf3-8bbb-3f72b8710763, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_ha_Disk1 was initiated on VM vm_ha by admin. >2014-08-21 11:06:29,800 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-6) [2cd61457] BaseAsyncTask::startPollingTask: Starting to poll task ed0aef1f-d35a-45a8-88af-cb265e908409. >2014-08-21 11:06:36,429 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-23) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-08-21 11:06:36,439 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) SPMAsyncTask::PollTask: Polling task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:36,444 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) BaseAsyncTask::OnTaskEndSuccess: Task ed0aef1f-d35a-45a8-88af-cb265e908409 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:36,445 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::EndActionIfNecessary: All tasks of command c15aff19-5992-40de-9c84-1e24980ea3d7 has ended -> executing endAction >2014-08-21 11:06:36,446 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: c15aff19-5992-40de-9c84-1e24980ea3d7): calling endAction . >2014-08-21 11:06:36,447 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:06:36,456 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-18) [disks_create_2b55890d-1f35-46db] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:06:36,500 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:06:36,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, imageId = de7bf293-5790-426c-9f8a-c391b1396c8a), log id: 5a005097 >2014-08-21 11:06:36,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [2cd61457] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8c694cf4, log id: 5a005097 >2014-08-21 11:06:36,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: disks_create_2b55890d-1f35-46db, Call Stack: null, Custom Event ID: -1, Message: The disk vm_ha_Disk1 was successfully added to VM vm_ha. >2014-08-21 11:06:36,668 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:06:36,669 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:06:36,675 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-18) SPMAsyncTask::ClearAsyncTask: Attempting to clear task ed0aef1f-d35a-45a8-88af-cb265e908409 >2014-08-21 11:06:36,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = ed0aef1f-d35a-45a8-88af-cb265e908409), log id: 5dfb6108 >2014-08-21 11:06:36,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=ed0aef1f-d35a-45a8-88af-cb265e908409), log id: 229f846c >2014-08-21 11:06:36,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, HSMClearTaskVDSCommand, log id: 229f846c >2014-08-21 11:06:36,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, SPMClearTaskVDSCommand, log id: 5dfb6108 >2014-08-21 11:06:36,745 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-18) BaseAsyncTask::RemoveTaskFromDB: Removed task ed0aef1f-d35a-45a8-88af-cb265e908409 from DataBase >2014-08-21 11:06:36,745 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-18) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity c15aff19-5992-40de-9c84-1e24980ea3d7 >2014-08-21 11:06:40,318 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:06:40,347 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, IsVmDuringInitiatingVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b), log id: 34bd7e7e >2014-08-21 11:06:40,347 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 34bd7e7e >2014-08-21 11:06:40,407 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Running command: RunVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:06:40,524 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@dc02e6ca), log id: ec8e3b0 >2014-08-21 11:06:40,527 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, UpdateVmDynamicDataVDSCommand, log id: ec8e3b0 >2014-08-21 11:06:40,579 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 68717012 >2014-08-21 11:06:40,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 7c00a846 >2014-08-21 11:06:40,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiYjQ2OTA2NjQtZDEyOC00ZDli >LWFhYWItNGYxZjM3MzIzNzVlIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=ac16ab88-e733-4373-bc4f-0dbec2b424b6}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=4efe0fe5-f86d-4359-a0b8-f6a1cc19feb4}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=080970ae-d7bb-494c-a4fd-832420935d2a}, {shared=false, volumeID=de7bf293-5790-426c-9f8a-c391b1396c8a, index=0, iface=virtio, imageID=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=5db0d96e-73dc-4370-a7fd-9b8f19330e3a}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=c15b2bc7-299d-4376-b060-fcf9ee1bca2c}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=4493feb7-855b-4bb5-9dd3-828324951568}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:06:40,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, CreateVmFromCloudInitVDSCommand, log id: 7c00a846 >2014-08-21 11:06:40,775 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 68717012 >2014-08-21 11:06:40,776 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:06:40,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [vms_syncAction_2ed08e1f-1103-44d1] Correlation ID: vms_syncAction_2ed08e1f-1103-44d1, Job ID: 9cff69db-1b74-461c-b66e-b954267899cf, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:06:42,840 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from WaitForLaunch --> PoweringUp >2014-08-21 11:06:42,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-44) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[04545787-d6d5-4b4d-a06f-2d42db7c933b]), log id: 6f4cacd0 >2014-08-21 11:06:42,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-44) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=4695, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@12e5c4c6, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 6f4cacd0 >2014-08-21 11:06:42,853 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Received a spice Device without an address when processing VM 04545787-d6d5-4b4d-a06f-2d42db7c933b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:06:52,125 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Lock Acquired to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:52,238 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups, ID: 0747ab80-2e72-4658-978a-a102f376cce8 Type: VdsGroups >2014-08-21 11:06:52,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Correlation ID: vms_create_64d6c1c7-12b0-43f6, Job ID: a6d4d145-c47d-452f-9990-30569e8ce4b2, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was created by admin. >2014-08-21 11:06:52,314 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [vms_create_64d6c1c7-12b0-43f6] Lock freed to object EngineLock [exclusiveLocks= key: vm_2 value: VM_NAME >, sharedLocks= ] >2014-08-21 11:06:53,176 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-5) [nics_create_906646f7-ed97-489e] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM, ID: 932ac343-a2d9-490a-a123-4e50109b7bee Type: VnicProfile >2014-08-21 11:06:53,221 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-5) [7293acbc] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:06:53,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7293acbc] Correlation ID: 7293acbc, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM vm_2. (User: admin) >2014-08-21 11:06:53,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7293acbc] Correlation ID: nics_create_906646f7-ed97-489e, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM vm_2. (User: admin) >2014-08-21 11:06:53,709 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM_DISK_BOOT >, sharedLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >] >2014-08-21 11:06:53,722 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Running command: AddDiskCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM, ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:53,724 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp--127.0.0.1-8702-7) [disks_create_04957bf8-6fd6-4ba3] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is vm_2_Disk1 >2014-08-21 11:06:53,766 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [59c24efb] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 59645b6f-d70b-45fb-aa66-c582c373d58a Type: Storage >2014-08-21 11:06:53,805 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [59c24efb] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM_DISK_BOOT >, sharedLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >] >2014-08-21 11:06:53,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] START, CreateImageVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, imageSizeInBytes = 2147483648, volumeFormat = RAW, newImageId = a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, newImageDescription = ), log id: 22f19931 >2014-08-21 11:06:53,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-08-21 11:06:53,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [59c24efb] FINISH, CreateImageVDSCommand, return: a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, log id: 22f19931 >2014-08-21 11:06:53,968 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [59c24efb] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 51674dc7-7774-4c78-8bb6-301f046ea07b >2014-08-21 11:06:53,980 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [59c24efb] CommandMultiAsyncTasks::AttachTask: Attaching task 73c605d7-0794-4766-ad42-b045d5bc24a8 to command 51674dc7-7774-4c78-8bb6-301f046ea07b. >2014-08-21 11:06:54,043 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [59c24efb] Adding task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:06:54,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [59c24efb] Correlation ID: disks_create_04957bf8-6fd6-4ba3, Job ID: 60495144-0d4b-43a9-8957-9e0352bf0332, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of vm_2_Disk1 was initiated on VM vm_2 by admin. >2014-08-21 11:06:54,063 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) [59c24efb] BaseAsyncTask::startPollingTask: Starting to poll task 73c605d7-0794-4766-ad42-b045d5bc24a8. >2014-08-21 11:06:56,448 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-88) Polling and updating Async Tasks: 4 tasks, 1 tasks to poll now >2014-08-21 11:06:56,482 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-88) SPMAsyncTask::PollTask: Polling task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:06:56,508 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-88) BaseAsyncTask::OnTaskEndSuccess: Task 73c605d7-0794-4766-ad42-b045d5bc24a8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:06:56,510 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-88) CommandAsyncTask::EndActionIfNecessary: All tasks of command 51674dc7-7774-4c78-8bb6-301f046ea07b has ended -> executing endAction >2014-08-21 11:06:56,511 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-88) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 51674dc7-7774-4c78-8bb6-301f046ea07b): calling endAction . >2014-08-21 11:06:56,512 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-08-21 11:06:56,527 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-33) [disks_create_04957bf8-6fd6-4ba3] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-08-21 11:06:56,550 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-08-21 11:06:56,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] START, GetImageInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, imageId = a95ef6a4-d3a2-421f-aa63-0bfaa543fd86), log id: 5cb8d46d >2014-08-21 11:06:56,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [59c24efb] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@ae15ce5a, log id: 5cb8d46d >2014-08-21 11:06:56,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) Correlation ID: disks_create_04957bf8-6fd6-4ba3, Call Stack: null, Custom Event ID: -1, Message: The disk vm_2_Disk1 was successfully added to VM vm_2. >2014-08-21 11:06:56,609 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-08-21 11:06:56,609 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-08-21 11:06:56,615 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 73c605d7-0794-4766-ad42-b045d5bc24a8 >2014-08-21 11:06:56,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 73c605d7-0794-4766-ad42-b045d5bc24a8), log id: 7161f5fe >2014-08-21 11:06:56,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=73c605d7-0794-4766-ad42-b045d5bc24a8), log id: 565d30d9 >2014-08-21 11:06:56,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMClearTaskVDSCommand, log id: 565d30d9 >2014-08-21 11:06:56,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMClearTaskVDSCommand, log id: 7161f5fe >2014-08-21 11:06:56,705 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-33) BaseAsyncTask::RemoveTaskFromDB: Removed task 73c605d7-0794-4766-ad42-b045d5bc24a8 from DataBase >2014-08-21 11:06:56,706 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-33) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 51674dc7-7774-4c78-8bb6-301f046ea07b >2014-08-21 11:07:04,626 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:07:04,718 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, IsVmDuringInitiatingVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae), log id: 4e1e9295 >2014-08-21 11:07:04,719 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4e1e9295 >2014-08-21 11:07:04,774 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Running command: RunVmCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:07:04,908 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, UpdateVmDynamicDataVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@8540b969), log id: 2fc77a89 >2014-08-21 11:07:04,911 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2fc77a89 >2014-08-21 11:07:04,944 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, CreateVmVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, vm=VM [vm_2]), log id: 3443d9df >2014-08-21 11:07:05,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] START, CreateVmFromCloudInitVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, vm=VM [vm_2]), log id: 2a17474d >2014-08-21 11:07:05,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiZmJjMGM0ZjAtMjRmOS00NWNj >LTgwMTEtMGJjYTllMmYyNGI2IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=2aba7272-a8c0-4106-9c5f-11d71d123617}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=188a64c6-c5ad-4f41-8b45-90ba7c9fb221}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=b83e7afd-5557-456a-9d5c-24c0bcaf53cc}, {shared=false, volumeID=a95ef6a4-d3a2-421f-aa63-0bfaa543fd86, index=0, iface=virtio, imageID=e346cbb0-7ed5-4c80-a189-b02706afc39c, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, format=raw, type=disk, readonly=false, deviceId=e346cbb0-7ed5-4c80-a189-b02706afc39c}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=007e6004-52e2-4a7c-896a-e0870bbe4b70}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=4d44beb2-d497-4b81-94ae-3dc402c8ce4f}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=f22ced93-a7cd-47ce-91a2-09f5ba1c5b17}],vmName=vm_2,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:07:05,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, CreateVmFromCloudInitVDSCommand, log id: 2a17474d >2014-08-21 11:07:05,049 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 3443d9df >2014-08-21 11:07:05,050 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:07:05,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_5f3e4580-66f2-49d4] Correlation ID: vms_syncAction_5f3e4580-66f2-49d4, Job ID: f00d9c7a-4620-4535-8f3c-73efc583c419, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was started by admin (Host: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:07:07,745 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) VM vm_2 61b86c61-6e16-439f-8fcd-486a2448e2ae moved from WaitForLaunch --> PoweringUp >2014-08-21 11:07:07,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) START, FullListVdsCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da], vmIds=[61b86c61-6e16-439f-8fcd-486a2448e2ae]), log id: 41ce4a52 >2014-08-21 11:07:07,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.97.75, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=61b86c61-6e16-439f-8fcd-486a2448e2ae, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=4860, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@4e716f3e, vmName=vm_2, fileTransferEnable=true, cpuType=Conroe}], log id: 41ce4a52 >2014-08-21 11:07:07,769 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a spice Device without an address when processing VM 61b86c61-6e16-439f-8fcd-486a2448e2ae devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.97.75}, device=spice, tlsPort=5903, type=graphics} >2014-08-21 11:07:36,205 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) Setting new tasks map. The map contains now 2 tasks >2014-08-21 11:08:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:08:06,207 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:10:17,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-58) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 11:10:17,210 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:17,211 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-21 11:10:17,227 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-21 11:10:22,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:22,479 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:22,480 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:22,501 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $61 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:27,633 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = Unknown, exit status = Normal), log id: 7e72464a >2014-08-21 11:10:27,646 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SetVmStatusVDSCommand, log id: 7e72464a >2014-08-21 11:10:27,684 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was set to the Unknown status. >2014-08-21 11:10:27,685 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = Unknown, exit status = Normal), log id: 5465096e >2014-08-21 11:10:27,690 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SetVmStatusVDSCommand, log id: 5465096e >2014-08-21 11:10:27,727 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was set to the Unknown status. >2014-08-21 11:10:27,728 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Server failed to respond, vds_id = a8fdd580-c752-4537-9cae-e49e468db7da, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 2, spm_status = None, non-responsive_timeout (seconds) = 61, error = java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:27,773 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-08-21 11:10:27,791 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-44) ResourceManager::vdsNotResponding entered for Host a8fdd580-c752-4537-9cae-e49e468db7da, rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:27,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:27,870 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:10:28,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:10:28,006 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:28,038 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:10:28,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 73e311b6 >2014-08-21 11:10:28,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 73e311b6 >2014-08-21 11:10:28,636 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:10:32,889 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:32,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:32,896 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:37,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:37,985 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:38,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:43,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:43,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:43,072 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:48,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) [43495829] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:48,143 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) [43495829] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:48,148 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) [43495829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:53,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:53,215 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [70db29ad] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:53,222 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [70db29ad] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:10:58,282 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:10:58,284 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:10:58,290 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:03,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:03,351 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:03,358 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:08,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:08,455 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:08,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:13,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:13,530 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:13,535 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:18,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:18,599 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:18,604 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:23,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:23,686 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:23,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:28,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:28,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:28,762 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:31,696 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-44) [74c29a42] SSH Soft Fencing command failed on host rose07.qa.lab.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-21 11:11:31,875 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:11:31,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:11:31,982 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:11:32,009 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:11:32,095 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:11:33,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:33,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:33,881 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:38,993 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:38,994 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:39,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:44,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:44,089 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:44,094 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:49,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:49,156 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:49,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:54,230 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:54,232 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:54,244 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:11:59,307 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:11:59,309 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:11:59,322 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:02,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Correlation ID: 17e68cb, Job ID: a399c5af-2ae7-436e-b214-2c36c691fab2, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-08-21 11:12:02,167 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:02,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:02,268 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:02,317 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [17e68cb] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:02,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 63793b05 >2014-08-21 11:12:02,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [17e68cb] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 63793b05 >2014-08-21 11:12:02,576 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:02,582 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:02,613 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 715d0a4e >2014-08-21 11:12:04,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:04,431 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $122 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:04,436 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:04,461 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, SetVdsStatusVDSCommand, log id: 715d0a4e >2014-08-21 11:12:04,570 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:04,571 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:04,602 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:04,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d799cca >2014-08-21 11:12:04,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, FenceVdsVDSCommand, return: Done, log id: 3d799cca >2014-08-21 11:12:04,744 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:12:09,745 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:12:09,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:09,823 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:09,857 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:09,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4d08de01 >2014-08-21 11:12:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 4d08de01 >2014-08-21 11:12:10,025 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:12:10,026 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = Down, exit status = Error), log id: 4a44a190 >2014-08-21 11:12:10,033 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24ef6141] FINISH, SetVmStatusVDSCommand, log id: 4a44a190 >2014-08-21 11:12:10,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24ef6141] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_ha was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:12:10,080 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:12:10,095 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = Down, exit status = Error), log id: 4103c53b >2014-08-21 11:12:10,100 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) [258c6729] FINISH, SetVmStatusVDSCommand, log id: 4103c53b >2014-08-21 11:12:10,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [258c6729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Vm vm_2 was shut down due to rose07.qa.lab.tlv.redhat.com host reboot or manual fence >2014-08-21 11:12:10,143 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:12:10,191 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 4752060e >2014-08-21 11:12:10,195 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-8-thread-44) [33e6082a] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 4752060e >2014-08-21 11:12:10,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [33e6082a] Correlation ID: 24ef6141, Job ID: 9c3cf8e6-bb89-45a4-b22e-76a435432fb8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-08-21 11:12:10,286 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-44) [5b771b1f] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:10,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:10,443 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:10,451 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:12:10,482 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b9f5bef >2014-08-21 11:12:10,487 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, SetVdsStatusVDSCommand, log id: 2b9f5bef >2014-08-21 11:12:10,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:10,660 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:10,720 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:10,722 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, IsVmDuringInitiatingVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b), log id: 74565e4d >2014-08-21 11:12:10,723 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 74565e4d >2014-08-21 11:12:10,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4d7ee243 >2014-08-21 11:12:10,743 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] Running command: RunVmCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:12:10,825 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, UpdateVmDynamicDataVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@1bba8144), log id: 39175f0c >2014-08-21 11:12:10,850 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, UpdateVmDynamicDataVDSCommand, log id: 39175f0c >2014-08-21 11:12:10,854 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, CreateVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 39c0b42a >2014-08-21 11:12:10,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, CreateVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, vm=VM [vm_ha]), log id: 5125dfe0 >2014-08-21 11:12:10,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, FenceVdsVDSCommand, return: Done, log id: 4d7ee243 >2014-08-21 11:12:10,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=4efe0fe5-f86d-4359-a0b8-f6a1cc19feb4}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=080970ae-d7bb-494c-a4fd-832420935d2a}, {shared=false, index=0, volumeID=de7bf293-5790-426c-9f8a-c391b1396c8a, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=59645b6f-d70b-45fb-aa66-c582c373d58a, imageID=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, specParams={}, optional=false, device=disk, poolID=ab2440ae-4286-4465-beed-c2ccc8256f8b, readonly=false, deviceId=37e9a63b-3e7e-48b9-bf08-fbb7480f90a1}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:fb:aa:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=5db0d96e-73dc-4370-a7fd-9b8f19330e3a}, {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=c15b2bc7-299d-4376-b060-fcf9ee1bca2c}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=4493feb7-855b-4bb5-9dd3-828324951568}],vmName=vm_ha,cpuType=Conroe,fileTransferEnable=true >2014-08-21 11:12:10,932 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:12:10,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, CreateVDSCommand, log id: 5125dfe0 >2014-08-21 11:12:11,007 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 39c0b42a >2014-08-21 11:12:11,008 INFO [org.ovirt.engine.core.bll.RunVmCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:12:11,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [4256016c] Correlation ID: 4256016c, Job ID: dd25a8fa-dd86-4745-895c-bcf72cfe3a6a, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:12:12,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:12,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:12,609 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:15,301 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from WaitForLaunch --> PoweringUp >2014-08-21 11:12:15,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-75) START, FullListVdsCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb], vmIds=[04545787-d6d5-4b4d-a06f-2d42db7c933b]), log id: 66a2483f >2014-08-21 11:12:15,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-75) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.2.45, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=17176, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@283554bd, vmName=vm_ha, fileTransferEnable=true, cpuType=Conroe}], log id: 66a2483f >2014-08-21 11:12:15,319 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Received a spice Device without an address when processing VM 04545787-d6d5-4b4d-a06f-2d42db7c933b devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.35.2.45}, device=spice, tlsPort=5901, type=graphics} >2014-08-21 11:12:15,933 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:12:15,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:12:16,000 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:12:16,031 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-44) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:12:16,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 54f3887e >2014-08-21 11:12:16,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 54f3887e >2014-08-21 11:12:16,200 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-44) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:12:16,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: 154fb5a1, Job ID: c8caa1b6-3deb-4496-b3a8-0afb45c17926, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-08-21 11:12:16,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) Correlation ID: 17e68cb, Job ID: f4acc64f-c0e3-4e74-8d85-cfe2e10399bd, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-08-21 11:12:16,224 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-44) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:12:17,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:17,691 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:17,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:22,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:22,757 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:22,766 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:27,855 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:27,856 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:27,894 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:32,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:32,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:32,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:38,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:38,085 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:38,098 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:43,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:43,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:43,165 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:48,239 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:48,241 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:48,246 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:53,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:53,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:53,315 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:12:58,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:12:58,394 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:12:58,399 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [799af561] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:03,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:03,460 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:03,466 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:08,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:08,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:08,586 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:12,995 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) VM vm_ha 04545787-d6d5-4b4d-a06f-2d42db7c933b moved from PoweringUp --> Up >2014-08-21 11:13:13,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: 4256016c, Job ID: dd25a8fa-dd86-4745-895c-bcf72cfe3a6a, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was restarted on Host monique-vds01.tlv.redhat.com >2014-08-21 11:13:13,013 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-30) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:13:13,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:13,652 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:13,658 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:18,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:18,718 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:18,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:23,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:23,795 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:23,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:28,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:28,863 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:28,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:33,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:33,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:33,936 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:38,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 11:13:38,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:13:39,007 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:13:42,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 48f00df7 >2014-08-21 11:13:42,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GetHardwareInfoVDSCommand, log id: 48f00df7 >2014-08-21 11:13:42,682 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [5dcc5fc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:42,690 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [38cc3fee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:13:42,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:13:42,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 30646a5f >2014-08-21 11:13:42,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 30646a5f >2014-08-21 11:13:42,973 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:13:43,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 23690ebf >2014-08-21 11:13:43,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 23690ebf >2014-08-21 11:13:43,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:43,347 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) [60da23ba] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:13:43,384 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-43) [60da23ba] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:13:43,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 26908e6d >2014-08-21 11:13:43,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 26908e6d >2014-08-21 11:13:43,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 638d6e0f >2014-08-21 11:13:43,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [60da23ba] FINISH, SetMOMPolicyParametersVDSCommand, log id: 638d6e0f >2014-08-21 11:13:43,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:13:43,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60da23ba] Correlation ID: 38cc3fee, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:13:43,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [428627dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:45,786 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,787 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,788 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_7432283e-79e5-4f24] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:45,794 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,795 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:45,851 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:45,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_7432283e-79e5-4f24, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:13:46,185 INFO [org.ovirt.engine.core.bll.StopVmCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] Running command: StopVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:46,189 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] START, DestroyVmVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, force=false, secondsToWait=0, gracefully=false, reason=), log id: 374138b8 >2014-08-21 11:13:46,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] START, DestroyVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vmId=04545787-d6d5-4b4d-a06f-2d42db7c933b, force=false, secondsToWait=0, gracefully=false, reason=), log id: 35065f7f >2014-08-21 11:13:46,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] FINISH, DestroyVDSCommand, log id: 35065f7f >2014-08-21 11:13:46,695 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] FINISH, DestroyVmVDSCommand, return: Down, log id: 374138b8 >2014-08-21 11:13:46,703 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-27) [2b4fe416] Running command: ProcessDownVmCommand internal: true. >2014-08-21 11:13:46,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [vms_syncAction_350fd20a-0c01-48b2] Correlation ID: vms_syncAction_350fd20a-0c01-48b2, Job ID: 565ad661-bfbf-4359-a769-e193a942ad30, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha powered off by admin (Host: monique-vds01.tlv.redhat.com) (Reason: Not Specified). >2014-08-21 11:13:46,981 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Lock Acquired to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:13:46,998 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Running command: RemoveVmCommand internal: false. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:46,999 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] START, SetVmStatusVDSCommand( vmId = 04545787-d6d5-4b4d-a06f-2d42db7c933b, status = ImageLocked, exit status = Normal), log id: 14bd70ee >2014-08-21 11:13:47,002 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] FINISH, SetVmStatusVDSCommand, log id: 14bd70ee >2014-08-21 11:13:47,004 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-3) [vms_delete_2628eafd-a5f7-40af] Lock freed to object EngineLock [exclusiveLocks= key: 04545787-d6d5-4b4d-a06f-2d42db7c933b value: VM >, sharedLocks= ] >2014-08-21 11:13:47,015 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-3) [25a5ba8e] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 04545787-d6d5-4b4d-a06f-2d42db7c933b Type: VM >2014-08-21 11:13:47,019 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:13:47,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = 37e9a63b-3e7e-48b9-bf08-fbb7480f90a1, postZeros = false, forceDelete = false), log id: 2db25ab7 >2014-08-21 11:13:47,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-3) [5c92cdab] FINISH, DeleteImageGroupVDSCommand, log id: 2db25ab7 >2014-08-21 11:13:47,198 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [5c92cdab] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 71a3406c-1e85-462a-a223-f29c4635425c >2014-08-21 11:13:47,199 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [5c92cdab] CommandMultiAsyncTasks::AttachTask: Attaching task 08748083-1564-4d95-9a52-7c0702981579 to command 71a3406c-1e85-462a-a223-f29c4635425c. >2014-08-21 11:13:47,259 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [5c92cdab] Adding task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:13:47,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5c92cdab] Correlation ID: vms_delete_2628eafd-a5f7-40af, Job ID: eb72cf33-8b2e-401a-9a2a-9fd2716b6aa4, Call Stack: null, Custom Event ID: -1, Message: VM vm_ha was successfully removed. >2014-08-21 11:13:47,361 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [5c92cdab] BaseAsyncTask::startPollingTask: Starting to poll task 08748083-1564-4d95-9a52-7c0702981579. >2014-08-21 11:13:47,600 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Lock Acquired to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:13:47,649 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Running command: RemoveVmCommand internal: false. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:13:47,650 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] START, SetVmStatusVDSCommand( vmId = 61b86c61-6e16-439f-8fcd-486a2448e2ae, status = ImageLocked, exit status = Normal), log id: 5c6605d0 >2014-08-21 11:13:47,654 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] FINISH, SetVmStatusVDSCommand, log id: 5c6605d0 >2014-08-21 11:13:47,669 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-5) [vms_delete_812df52f-aad8-4de7] Lock freed to object EngineLock [exclusiveLocks= key: 61b86c61-6e16-439f-8fcd-486a2448e2ae value: VM >, sharedLocks= ] >2014-08-21 11:13:47,692 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp--127.0.0.1-8702-5) [78764fe1] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 61b86c61-6e16-439f-8fcd-486a2448e2ae Type: VM >2014-08-21 11:13:47,695 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-08-21 11:13:47,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] START, DeleteImageGroupVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, storageDomainId = 59645b6f-d70b-45fb-aa66-c582c373d58a, imageGroupId = e346cbb0-7ed5-4c80-a189-b02706afc39c, postZeros = false, forceDelete = false), log id: 49de24e3 >2014-08-21 11:13:48,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp--127.0.0.1-8702-5) [1f31faa4] FINISH, DeleteImageGroupVDSCommand, log id: 49de24e3 >2014-08-21 11:13:48,121 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-5) [1f31faa4] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command e62513d1-c307-402a-9ffe-624682997e88 >2014-08-21 11:13:48,122 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-5) [1f31faa4] CommandMultiAsyncTasks::AttachTask: Attaching task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 to command e62513d1-c307-402a-9ffe-624682997e88. >2014-08-21 11:13:48,133 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-5) [1f31faa4] Adding task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-21 11:13:48,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [1f31faa4] Correlation ID: vms_delete_812df52f-aad8-4de7, Job ID: 03fc3aaf-eee1-40cf-97a4-e7f82e159751, Call Stack: null, Custom Event ID: -1, Message: VM vm_2 was successfully removed. >2014-08-21 11:13:48,241 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-5) [1f31faa4] BaseAsyncTask::startPollingTask: Starting to poll task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39. >2014-08-21 11:13:48,470 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,471 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,472 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_update_3209115f-f22d-46d6] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:13:48,527 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,527 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:13:48,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:13:48,739 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:13:48,772 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:13:48,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5d401652 >2014-08-21 11:13:48,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5d401652 >2014-08-21 11:13:48,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_update_3209115f-f22d-46d6, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:13:56,529 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-08-21 11:13:56,538 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:13:56,544 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) BaseAsyncTask::OnTaskEndSuccess: Task 08748083-1564-4d95-9a52-7c0702981579 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:13:56,545 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::EndActionIfNecessary: All tasks of command 71a3406c-1e85-462a-a223-f29c4635425c has ended -> executing endAction >2014-08-21 11:13:56,546 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 71a3406c-1e85-462a-a223-f29c4635425c): calling endAction . >2014-08-21 11:13:56,548 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-21 11:13:56,549 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:13:56,562 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:13:56,563 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:13:56,564 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:13:56,567 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) BaseAsyncTask::OnTaskEndSuccess: Task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-21 11:13:56,569 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::EndActionIfNecessary: All tasks of command e62513d1-c307-402a-9ffe-624682997e88 has ended -> executing endAction >2014-08-21 11:13:56,570 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-1) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: e62513d1-c307-402a-9ffe-624682997e88): calling endAction . >2014-08-21 11:13:56,571 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-08-21 11:13:56,583 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 08748083-1564-4d95-9a52-7c0702981579 >2014-08-21 11:13:56,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 08748083-1564-4d95-9a52-7c0702981579), log id: 68931c90 >2014-08-21 11:13:56,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=08748083-1564-4d95-9a52-7c0702981579), log id: 12386793 >2014-08-21 11:13:56,589 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-08-21 11:13:56,591 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-08-21 11:13:56,592 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-08-21 11:13:56,595 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 >2014-08-21 11:13:56,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] START, SPMClearTaskVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, taskId = 48a5c35c-a40f-4e17-80c5-f7b2c740cd39), log id: 6a938c58 >2014-08-21 11:13:56,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] FINISH, HSMClearTaskVDSCommand, log id: 12386793 >2014-08-21 11:13:56,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] FINISH, SPMClearTaskVDSCommand, log id: 68931c90 >2014-08-21 11:13:56,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=48a5c35c-a40f-4e17-80c5-f7b2c740cd39), log id: 6a2e9087 >2014-08-21 11:13:56,625 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] BaseAsyncTask::RemoveTaskFromDB: Removed task 08748083-1564-4d95-9a52-7c0702981579 from DataBase >2014-08-21 11:13:56,626 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-17) [vms_delete_2628eafd-a5f7-40af] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 71a3406c-1e85-462a-a223-f29c4635425c >2014-08-21 11:13:56,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] FINISH, HSMClearTaskVDSCommand, log id: 6a2e9087 >2014-08-21 11:13:56,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] FINISH, SPMClearTaskVDSCommand, log id: 6a938c58 >2014-08-21 11:13:56,693 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] BaseAsyncTask::RemoveTaskFromDB: Removed task 48a5c35c-a40f-4e17-80c5-f7b2c740cd39 from DataBase >2014-08-21 11:13:56,694 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-10) [vms_delete_812df52f-aad8-4de7] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity e62513d1-c307-402a-9ffe-624682997e88 >2014-08-21 11:15:06,210 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-36) Setting new tasks map. The map contains now 0 tasks >2014-08-21 11:15:06,211 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-36) Cleared all tasks of pool ab2440ae-4286-4465-beed-c2ccc8256f8b. >2014-08-21 11:15:16,945 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_d5c69dac-42c6-4222] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:17,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:17,076 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:17,077 ERROR [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) VDS status for vds a8fdd580-c752-4537-9cae-e49e468db7da:rose07.qa.lab.tlv.redhat.com is Up >2014-08-21 11:15:17,078 WARN [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) CanDoAction of action StartVds failed. Reasons:VAR__ACTION__START,VDS_STATUS_NOT_VALID_FOR_START >2014-08-21 11:15:17,079 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:17,085 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-3) Operation Failed: [Cannot perform Start operation, Host has to be in one of the following statuses: Down ,Non Responsive or Maintenance.] >2014-08-21 11:15:20,091 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,092 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,093 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_5d901f2e-f132-4c9a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:20,103 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,104 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,128 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:20,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_5d901f2e-f132-4c9a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:15:20,325 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,326 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,327 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_67fc0a0b-2fa5-42c3] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:20,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,368 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:15:20,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:20,598 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:20,629 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:20,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 75bcfa45 >2014-08-21 11:15:20,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 75bcfa45 >2014-08-21 11:15:20,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_67fc0a0b-2fa5-42c3, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:15:21,413 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:21,444 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 5a7ce233 >2014-08-21 11:15:21,449 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] FINISH, SetVdsStatusVDSCommand, log id: 5a7ce233 >2014-08-21 11:15:21,525 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:21,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [hosts_syncAction_bdf851e1-0229-49a1] Correlation ID: hosts_syncAction_bdf851e1-0229-49a1, Job ID: b08da44b-0cab-459e-8809-b3bd0d09fe35, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-21 11:15:21,828 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:21,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:21,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Removing vds [a8fdd580-c752-4537-9cae-e49e468db7da] from the domain in maintenance cache >2014-08-21 11:15:21,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Removing host(s) [a8fdd580-c752-4537-9cae-e49e468db7da] from hosts unseen domain report cache >2014-08-21 11:15:21,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 2), log id: 5791ac6f >2014-08-21 11:15:24,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, DisconnectStoragePoolVDSCommand, log id: 5791ac6f >2014-08-21 11:15:24,187 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:15:24,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4888b55 >2014-08-21 11:15:24,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4888b55 >2014-08-21 11:15:25,157 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_syncAction_c8fcdef3-2703-4847] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:15:25,261 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:25,263 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:25,267 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Running command: StopVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:15:25,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Down, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 42c55478 >2014-08-21 11:15:25,302 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, SetVdsStatusVDSCommand, log id: 42c55478 >2014-08-21 11:15:25,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:25,416 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:25,447 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:25,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 190495ec >2014-08-21 11:15:25,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Done, log id: 190495ec >2014-08-21 11:15:25,573 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:15:30,574 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:15:30,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:15:30,640 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:15:30,671 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:15:30,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 67d557a8 >2014-08-21 11:15:30,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 67d557a8 >2014-08-21 11:15:30,805 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:15:30,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_syncAction_c8fcdef3-2703-4847, Job ID: 4e422f17-35fb-4192-9896-fbb25d58c78d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:15:30,813 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:19:15,499 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_122609fa-80a2-4ec3] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:19:15,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:15,678 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:15,684 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Running command: StartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:19:15,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 16972639 >2014-08-21 11:19:15,721 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, SetVdsStatusVDSCommand, log id: 16972639 >2014-08-21 11:19:15,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:15,818 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:15,849 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:19:15,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5c2a1792 >2014-08-21 11:19:15,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Done, log id: 5c2a1792 >2014-08-21 11:19:15,983 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:19:20,984 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:19:21,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:19:21,055 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:19:21,087 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:19:21,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 7ea486be >2014-08-21 11:19:21,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7ea486be >2014-08-21 11:19:21,244 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:19:21,275 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_122609fa-80a2-4ec3, Job ID: 2a31d4c9-4922-4ea7-8f60-e102fccbbbaa, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:19:21,280 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:21:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:21:00,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:21:00,709 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:21:04,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 51a19d95 >2014-08-21 11:21:04,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [1dae0525] FINISH, GetHardwareInfoVDSCommand, log id: 51a19d95 >2014-08-21 11:21:04,562 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [439ab763] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:21:04,571 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [43d2e940] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:21:04,577 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:21:04,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 29c066f5 >2014-08-21 11:21:04,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 29c066f5 >2014-08-21 11:21:04,849 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:21:04,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 5a4350bb >2014-08-21 11:21:05,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, ConnectStoragePoolVDSCommand, log id: 5a4350bb >2014-08-21 11:21:05,168 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:21:05,169 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) [605a093a] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:21:05,200 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-89) [605a093a] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:21:05,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: ee485f4 >2014-08-21 11:21:05,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: ee485f4 >2014-08-21 11:21:05,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 570fa438 >2014-08-21 11:21:05,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [605a093a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 570fa438 >2014-08-21 11:21:05,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:21:05,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [605a093a] Correlation ID: 43d2e940, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:21:06,081 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [2bdb3d72] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,667 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,668 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,669 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_d25ce43b-f8f3-41a2] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,679 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,679 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,740 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:22,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_d25ce43b-f8f3-41a2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:22:22,899 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,900 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,901 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_19374966-cd46-42d1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:22,948 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:22,948 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:22:23,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:23,199 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:23,229 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:23,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 475d5de2 >2014-08-21 11:22:23,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 475d5de2 >2014-08-21 11:22:23,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_19374966-cd46-42d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:22:26,309 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_3b20c00a-dbe1-4242] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:22:26,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,414 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,419 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:26,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,554 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:26,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2c8bd298 >2014-08-21 11:22:26,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2c8bd298 >2014-08-21 11:22:26,805 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:26,836 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 14acf354 >2014-08-21 11:22:26,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, SetVdsStatusVDSCommand, log id: 14acf354 >2014-08-21 11:22:26,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:26,938 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:26,969 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:26,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 77e085ac >2014-08-21 11:22:27,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, FenceVdsVDSCommand, return: Done, log id: 77e085ac >2014-08-21 11:22:27,094 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:22:32,095 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:22:32,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,165 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,202 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [505bca26] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:32,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5b0f5c43 >2014-08-21 11:22:32,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [505bca26] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 5b0f5c43 >2014-08-21 11:22:32,394 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [505bca26] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:22:32,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [505bca26] Correlation ID: 505bca26, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:22:32,473 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [1d0f49a] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:32,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,610 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,627 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:22:32,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 430da4bb >2014-08-21 11:22:32,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 430da4bb >2014-08-21 11:22:32,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:32,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:32,819 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:32,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 71aa9360 >2014-08-21 11:22:32,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 71aa9360 >2014-08-21 11:22:32,982 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:22:37,983 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:22:38,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:22:38,064 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:22:38,095 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:22:38,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 5319f5f2 >2014-08-21 11:22:38,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5319f5f2 >2014-08-21 11:22:38,235 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:22:38,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: fd10dc0, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:22:38,244 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_3b20c00a-dbe1-4242, Job ID: 57d35524-28e8-4634-8c89-d63f80b68145, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:22:38,247 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:24:17,450 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:24:17,452 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:24:17,453 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:24:20,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 72064b80 >2014-08-21 11:24:21,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, GetHardwareInfoVDSCommand, log id: 72064b80 >2014-08-21 11:24:21,077 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [6e405299] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:24:21,086 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [2bd2395b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:24:21,092 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:24:21,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d8984d7 >2014-08-21 11:24:21,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: d8984d7 >2014-08-21 11:24:21,396 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:24:21,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 7904b840 >2014-08-21 11:24:21,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, ConnectStoragePoolVDSCommand, log id: 7904b840 >2014-08-21 11:24:21,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:24:21,751 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [6778b85a] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:24:21,782 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-24) [6778b85a] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:24:21,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 72d426cd >2014-08-21 11:24:21,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 72d426cd >2014-08-21 11:24:21,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 63a83ded >2014-08-21 11:24:21,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-24) [6778b85a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 63a83ded >2014-08-21 11:24:22,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:24:22,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6778b85a] Correlation ID: 2bd2395b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:24:22,074 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [36ee3909] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,241 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,242 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,243 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_af0cd597-1a75-4519] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,249 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,275 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:39,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_af0cd597-1a75-4519, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:25:39,468 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,468 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,469 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_df081405-3bb6-4c6e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:39,485 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,485 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:25:39,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:39,693 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:39,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-5) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:39,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 1716f2be >2014-08-21 11:25:39,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-5) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1716f2be >2014-08-21 11:25:39,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_df081405-3bb6-4c6e, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:25:40,457 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:40,488 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 69be510c >2014-08-21 11:25:40,493 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] FINISH, SetVdsStatusVDSCommand, log id: 69be510c >2014-08-21 11:25:40,566 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:40,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [hosts_syncAction_77934a77-f3b5-409c] Correlation ID: hosts_syncAction_77934a77-f3b5-409c, Job ID: a470e432-f725-4b83-bb2d-70aa22be7090, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-08-21 11:25:41,549 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:41,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Clearing cache of pool: ab2440ae-4286-4465-beed-c2ccc8256f8b for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:41,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing vds [a8fdd580-c752-4537-9cae-e49e468db7da] from the domain in maintenance cache >2014-08-21 11:25:41,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing host(s) [a8fdd580-c752-4537-9cae-e49e468db7da] from hosts unseen domain report cache >2014-08-21 11:25:41,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, vds_spm_id = 2), log id: 72d0406e >2014-08-21 11:25:44,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, DisconnectStoragePoolVDSCommand, log id: 72d0406e >2014-08-21 11:25:44,607 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:25:44,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1fc65a72 >2014-08-21 11:25:44,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, DisconnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 1fc65a72 >2014-08-21 11:25:53,794 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_syncAction_c7af6e83-3791-4592] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:25:53,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:53,952 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:53,959 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:54,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:54,056 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:54,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:54,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3adc6ca3 >2014-08-21 11:25:54,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3adc6ca3 >2014-08-21 11:25:54,263 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:25:54,299 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2d02b094 >2014-08-21 11:25:54,304 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, SetVdsStatusVDSCommand, log id: 2d02b094 >2014-08-21 11:25:54,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:54,409 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:54,440 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:54,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 63ed3213 >2014-08-21 11:25:54,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, FenceVdsVDSCommand, return: Done, log id: 63ed3213 >2014-08-21 11:25:54,575 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:25:59,575 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:25:59,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:25:59,646 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:25:59,677 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) [3a6f926c] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:25:59,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 372bffb5 >2014-08-21 11:25:59,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 372bffb5 >2014-08-21 11:25:59,807 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-6) [3a6f926c] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:25:59,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3a6f926c] Correlation ID: 3a6f926c, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:25:59,869 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-6) [4d99ff20] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:26:00,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:00,072 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:00,076 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:26:00,107 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7139e185 >2014-08-21 11:26:00,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, SetVdsStatusVDSCommand, log id: 7139e185 >2014-08-21 11:26:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:00,212 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:00,243 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:26:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 48085c2b >2014-08-21 11:26:00,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Done, log id: 48085c2b >2014-08-21 11:26:00,368 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:26:05,369 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:26:05,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:26:05,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:26:05,489 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-6) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:26:05,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6bbf1a1c >2014-08-21 11:26:06,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6bbf1a1c >2014-08-21 11:26:06,152 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-6) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:26:06,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 6275e16b, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:26:06,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_syncAction_c7af6e83-3791-4592, Job ID: d2d68337-0711-41be-a8dc-d781ea115aa4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:26:06,184 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-6) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:29:07,334 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-21 11:29:07,339 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Running command: ActivateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:07,340 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,341 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,379 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4e8560a9 >2014-08-21 11:29:07,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-4) [hosts_syncAction_c5d3d199-c3a6-4b84] FINISH, SetVdsStatusVDSCommand, log id: 4e8560a9 >2014-08-21 11:29:07,426 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-08-21 11:29:07,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_syncAction_c5d3d199-c3a6-4b84, Job ID: 8c951f28-66b4-4ee6-bcc4-35d63f948c31, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-08-21 11:29:07,452 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS >, sharedLocks= ] >2014-08-21 11:29:08,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:29:08,233 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:29:08,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:29:08,273 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-08-21 11:29:12,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 1fd96cec >2014-08-21 11:29:12,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GetHardwareInfoVDSCommand, log id: 1fd96cec >2014-08-21 11:29:12,405 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [20fe3322] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:12,413 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [472dd376] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:29:12,419 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:29:12,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 19e8d6d5 >2014-08-21 11:29:12,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 19e8d6d5 >2014-08-21 11:29:12,682 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:29:12,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: b05c29f >2014-08-21 11:29:12,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: b05c29f >2014-08-21 11:29:13,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:13,004 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [ef1a542] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:13,035 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [ef1a542] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:29:13,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 688e67c2 >2014-08-21 11:29:13,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 688e67c2 >2014-08-21 11:29:13,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 34769bc9 >2014-08-21 11:29:13,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [ef1a542] FINISH, SetMOMPolicyParametersVDSCommand, log id: 34769bc9 >2014-08-21 11:29:13,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:29:13,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [ef1a542] Correlation ID: 472dd376, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:29:13,324 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,065 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,066 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,067 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_c7b5b417-0474-4272] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,080 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,080 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,105 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:18,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_c7b5b417-0474-4272, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:29:18,365 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,366 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,367 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_208f0beb-c867-4bab] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:18,456 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,456 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:29:18,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:18,853 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:18,912 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:18,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 657bdccc >2014-08-21 11:29:19,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 657bdccc >2014-08-21 11:29:19,120 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:19,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1b3fcabe >2014-08-21 11:29:19,372 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:19,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1b3fcabe >2014-08-21 11:29:19,374 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:29:19,721 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:29:19,721 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:29:19,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:19,845 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:19,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:19,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 154fa521 >2014-08-21 11:29:20,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:20,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 154fa521 >2014-08-21 11:29:20,100 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:20,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_208f0beb-c867-4bab, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:29:23,600 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) [hosts_syncAction_2dd2b5e3-cb4f-4e77] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:29:23,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:23,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:23,847 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:24,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:24,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:24,114 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:24,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3badc02c >2014-08-21 11:29:24,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3badc02c >2014-08-21 11:29:24,318 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:24,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 723b7e84 >2014-08-21 11:29:24,509 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:24,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 723b7e84 >2014-08-21 11:29:24,511 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:29:24,762 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:29:24,763 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:29:25,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:25,059 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:25,060 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:slitmano, Options: >2014-08-21 11:29:25,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 573fe75c >2014-08-21 11:29:25,259 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:29:25,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 573fe75c >2014-08-21 11:29:25,353 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:25,429 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12dd188a >2014-08-21 11:29:25,433 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, SetVdsStatusVDSCommand, log id: 12dd188a >2014-08-21 11:29:25,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:25,650 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:25,746 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:25,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 238265f1 >2014-08-21 11:29:25,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, FenceVdsVDSCommand, return: Done, log id: 238265f1 >2014-08-21 11:29:25,999 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:29:31,000 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:29:31,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:31,127 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:31,187 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) [38b39414] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:31,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7a000d81 >2014-08-21 11:29:31,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) [38b39414] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 7a000d81 >2014-08-21 11:29:31,316 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-8) [38b39414] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:29:31,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [38b39414] Correlation ID: 38b39414, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:29:31,396 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-8) [7f50e682] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:31,664 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:31,666 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:31,671 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:29:31,752 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6e693b16 >2014-08-21 11:29:31,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, SetVdsStatusVDSCommand, log id: 6e693b16 >2014-08-21 11:29:32,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:32,010 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:32,088 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:32,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 53db00da >2014-08-21 11:29:32,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Done, log id: 53db00da >2014-08-21 11:29:32,259 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:29:37,260 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:29:37,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:29:37,397 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:29:37,468 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-8) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:29:37,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 5043ccef >2014-08-21 11:29:37,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-8) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5043ccef >2014-08-21 11:29:37,610 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-8) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:29:37,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: 12ea24eb, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:29:37,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: hosts_syncAction_2dd2b5e3-cb4f-4e77, Job ID: 653f4b3c-724f-473d-bf59-5b9b12f166ca, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:29:37,633 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-8) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:31:15,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:31:15,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:31:15,273 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:31:18,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 99cc92e >2014-08-21 11:31:18,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, GetHardwareInfoVDSCommand, log id: 99cc92e >2014-08-21 11:31:19,024 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [e46b1a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:31:19,032 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [7e63a6eb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:31:19,039 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:31:19,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2be9cbf3 >2014-08-21 11:31:19,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 2be9cbf3 >2014-08-21 11:31:19,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:31:19,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6a828d8d >2014-08-21 11:31:19,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 6a828d8d >2014-08-21 11:31:19,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:31:19,786 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:31:19,846 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 11:31:19,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 78299c3c >2014-08-21 11:31:20,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 78299c3c >2014-08-21 11:31:20,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 67304916 >2014-08-21 11:31:20,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [7aa13b76] FINISH, SetMOMPolicyParametersVDSCommand, log id: 67304916 >2014-08-21 11:31:20,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:31:20,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [7aa13b76] Correlation ID: 7e63a6eb, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:31:20,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:38,795 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,796 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,797 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_4b70da29-d815-447d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:38,805 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,806 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,830 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:38,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_4b70da29-d815-447d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:32:38,995 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,996 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:38,997 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-5) [hosts_update_236b845d-6a2a-4c00] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:39,012 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,013 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-5) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,048 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:39,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: hosts_update_236b845d-6a2a-4c00, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:32:39,409 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,410 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,411 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_accb1d5a-68e3-43c1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:32:39,437 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,438 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:32:39,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:32:39,810 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:32:39,868 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:32:39,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 43772bc9 >2014-08-21 11:33:20,072 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:33:20,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 43772bc9 >2014-08-21 11:33:20,075 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:33:20,314 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:33:20,315 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:33:20,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:33:20,458 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:33:20,459 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:33:20,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 199aa376 >2014-08-21 11:34:00,629 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:00,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 199aa376 >2014-08-21 11:34:00,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_accb1d5a-68e3-43c1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:34:03,540 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_5787e32d-62bf-4865] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:34:03,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:03,730 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:03,744 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:34:03,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:03,931 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:04,003 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:34:04,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 239a79a1 >2014-08-21 11:34:44,209 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:34:44,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 239a79a1 >2014-08-21 11:34:44,213 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:34:44,526 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:34:44,527 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:34:44,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:34:44,653 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:34:44,654 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:34:44,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4381621a >2014-08-21 11:35:24,928 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:35:24,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 4381621a >2014-08-21 11:35:25,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:35:25,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 26c3dbed >2014-08-21 11:35:25,276 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:35:25,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 26c3dbed >2014-08-21 11:35:25,278 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:35:25,643 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:35:25,644 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:35:25,964 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:25,965 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:25,966 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:35:25,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 1234e230 >2014-08-21 11:35:26,159 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:35:26,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 1234e230 >2014-08-21 11:35:26,306 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:35:26,409 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 60943cfe >2014-08-21 11:35:26,415 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, SetVdsStatusVDSCommand, log id: 60943cfe >2014-08-21 11:35:26,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:26,603 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:26,661 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:35:26,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: a1f55c9 >2014-08-21 11:35:46,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Done, log id: a1f55c9 >2014-08-21 11:35:46,808 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:35:51,809 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:35:51,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:35:51,931 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:35:51,990 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:35:51,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 48f57bce >2014-08-21 11:36:32,162 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:36:32,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 48f57bce >2014-08-21 11:36:32,165 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:36:32,407 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:36:32,408 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:36:32,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:36:32,530 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:36:32,531 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:36:32,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 28013745 >2014-08-21 11:37:12,712 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:37:12,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 28013745 >2014-08-21 11:37:12,716 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Host rose07.qa.lab.tlv.redhat.com Primary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-21 11:37:12,776 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-21 11:37:12,778 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-21 11:37:13,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:13,024 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:13,084 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:13,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 4e8d0da6 >2014-08-21 11:37:13,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4e8d0da6 >2014-08-21 11:37:13,245 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:37:18,246 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:37:18,398 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:18,399 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:18,459 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:18,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 2eafb3d2 >2014-08-21 11:37:18,690 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:37:18,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 2eafb3d2 >2014-08-21 11:37:18,692 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:37:18,940 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:37:18,941 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:37:19,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:19,071 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:19,072 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [4a55168e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:apc_snmp, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:10.35.23.24, User:slitmano, Options: >2014-08-21 11:37:19,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = 10.35.23.24, port = , type = apc_snmp, user = slitmano, password = ******, options = ''), log id: 3d0d68db >2014-08-21 11:37:19,287 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Failed: You have to enter plug number, Please use '-h' for usage] >2014-08-21 11:37:19,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, FenceVdsVDSCommand, return: Test Failed, [Failed: You have to enter plug number, Please use '-h' for usage], log id: 3d0d68db >2014-08-21 11:37:19,289 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Host rose07.qa.lab.tlv.redhat.com Secondary PM Agent definitions are corrupted, Waiting for Host to Stop aborted. >2014-08-21 11:37:19,350 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com Restart status, Please Restart Host rose07.qa.lab.tlv.redhat.com manually. >2014-08-21 11:37:19,351 ERROR [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Failed to verify host rose07.qa.lab.tlv.redhat.com stop status. Have retried 18 times with delay of 10 seconds between each retry. >2014-08-21 11:37:19,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to power fence host rose07.qa.lab.tlv.redhat.com. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >2014-08-21 11:37:19,417 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: 4a55168e, Job ID: a0886db3-1810-461b-afa2-71441cdc5bcc, Call Stack: null, Custom Event ID: -1, Message: Failed to stop Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-21 11:37:19,422 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Restart host action failed, updating host a8fdd580-c752-4537-9cae-e49e468db7da to NonResponsive >2014-08-21 11:37:19,483 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7ca3a618 >2014-08-21 11:37:19,487 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [4a55168e] FINISH, SetVdsStatusVDSCommand, log id: 7ca3a618 >2014-08-21 11:37:19,491 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4a55168e] Correlation ID: hosts_syncAction_5787e32d-62bf-4865, Job ID: a0886db3-1810-461b-afa2-71441cdc5bcc, Call Stack: null, Custom Event ID: -1, Message: Failed to restart Host rose07.qa.lab.tlv.redhat.com, (User: admin). >2014-08-21 11:37:19,494 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [4a55168e] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:37:19,495 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [] >2014-08-21 11:37:20,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 11011807 >2014-08-21 11:37:21,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4256016c] FINISH, GetHardwareInfoVDSCommand, log id: 11011807 >2014-08-21 11:37:21,124 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [2e6c4d38] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:37:21,133 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-79) [7ad48c4f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:37:21,138 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:37:21,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d08d0dd >2014-08-21 11:37:21,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: d08d0dd >2014-08-21 11:37:21,211 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:37:21,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 2131eb5e >2014-08-21 11:37:21,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 2131eb5e >2014-08-21 11:37:21,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:37:21,641 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:37:21,700 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:37:21,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: e50d277 >2014-08-21 11:38:01,881 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:01,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: e50d277 >2014-08-21 11:38:01,884 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 11:38:02,126 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-08-21 11:38:02,127 WARN [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 11:38:02,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:02,251 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:02,252 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-79) [353e8d77] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:blabla.blibli.com, User:root, Options: >2014-08-21 11:38:02,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blabla.blibli.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 45a4dd72 >2014-08-21 11:38:42,452 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:42,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds], log id: 45a4dd72 >2014-08-21 11:38:42,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: a90f2d5 >2014-08-21 11:38:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-79) [353e8d77] FINISH, SetMOMPolicyParametersVDSCommand, log id: a90f2d5 >2014-08-21 11:38:42,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:38:42,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blabla.blibli.com...Chassis power = Unknown, Failed, ipmilan: Failed to connect after 20 seconds] >2014-08-21 11:38:42,697 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [353e8d77] Correlation ID: 7ad48c4f, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Host rose07.qa.lab.tlv.redhat.com power management. >2014-08-21 11:38:42,765 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [351eedf6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,440 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,441 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,442 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_b4c960f5-713e-44fa] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,446 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,447 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,471 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:52,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_b4c960f5-713e-44fa, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:52,659 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,659 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,660 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_239a8a7a-e400-4521] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:52,685 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,685 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,708 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:52,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_239a8a7a-e400-4521, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:52,988 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,989 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:52,990 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_a31a3e51-df1e-410a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:53,015 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:53,015 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:38:53,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:53,222 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:53,254 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:53,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6a723d1d >2014-08-21 11:38:53,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6a723d1d >2014-08-21 11:38:53,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_a31a3e51-df1e-410a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:38:57,164 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_1c241878-146a-49cc] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:38:57,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:57,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:57,422 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:57,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:57,630 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:57,673 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:57,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4819b274 >2014-08-21 11:38:57,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4819b274 >2014-08-21 11:38:57,994 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:38:58,036 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 9e1c1d0 >2014-08-21 11:38:58,062 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, SetVdsStatusVDSCommand, log id: 9e1c1d0 >2014-08-21 11:38:58,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:38:58,325 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:38:58,356 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:38:58,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 29347f4d >2014-08-21 11:38:58,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, FenceVdsVDSCommand, return: Done, log id: 29347f4d >2014-08-21 11:38:58,503 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:39:03,504 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:39:03,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:03,572 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:03,602 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1838c425] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:03,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 423725d5 >2014-08-21 11:39:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1838c425] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 423725d5 >2014-08-21 11:39:03,737 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1838c425] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:39:03,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1838c425] Correlation ID: 1838c425, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:39:03,781 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-7) [411d0e1a] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:39:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:03,918 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:03,923 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:39:03,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6aa0553d >2014-08-21 11:39:03,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVdsStatusVDSCommand, log id: 6aa0553d >2014-08-21 11:39:04,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:04,054 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:04,085 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:04,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 2c2cac3e >2014-08-21 11:39:04,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Done, log id: 2c2cac3e >2014-08-21 11:39:04,217 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:39:09,218 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:39:09,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:39:09,292 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:39:09,334 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:39:09,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 47b0b71a >2014-08-21 11:39:09,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 47b0b71a >2014-08-21 11:39:09,472 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:39:09,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 55453975, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:39:09,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_syncAction_1c241878-146a-49cc, Job ID: 362576ef-3438-4aba-817a-a1466655db8c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:39:09,512 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:40:47,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:40:47,067 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:40:47,068 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:40:50,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 78823549 >2014-08-21 11:40:50,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [70db29ad] FINISH, GetHardwareInfoVDSCommand, log id: 78823549 >2014-08-21 11:40:50,825 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [598a33c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:40:50,836 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [2f31657c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:40:50,841 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [25697095] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:40:50,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 115261d >2014-08-21 11:40:51,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 115261d >2014-08-21 11:40:51,093 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [25697095] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:40:51,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 74d99b54 >2014-08-21 11:40:51,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 74d99b54 >2014-08-21 11:40:51,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:40:51,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [25697095] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:40:51,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-8) [25697095] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:40:51,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 134f0420 >2014-08-21 11:40:51,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 134f0420 >2014-08-21 11:40:51,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 63851b2d >2014-08-21 11:40:51,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [25697095] FINISH, SetMOMPolicyParametersVDSCommand, log id: 63851b2d >2014-08-21 11:40:51,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:40:51,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [25697095] Correlation ID: 2f31657c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:40:51,799 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [6c1dc07e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:10,667 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-8) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:42:12,839 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,840 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,840 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [hosts_update_e57e2441-5695-496f] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:12,844 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,845 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:12,870 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:12,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: hosts_update_e57e2441-5695-496f, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:42:13,096 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,097 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,098 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [hosts_update_9fee5846-4807-4217] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:13,144 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,145 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:42:13,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:13,346 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:13,376 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:13,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 760ebdec >2014-08-21 11:42:13,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 760ebdec >2014-08-21 11:42:13,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: hosts_update_9fee5846-4807-4217, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:42:16,667 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_syncAction_b264b449-6c7a-417a] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:42:16,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:16,771 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:16,775 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:16,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:16,883 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:16,913 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:16,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 25343c22 >2014-08-21 11:42:17,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 25343c22 >2014-08-21 11:42:17,085 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:17,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 76b8dcd >2014-08-21 11:42:17,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, SetVdsStatusVDSCommand, log id: 76b8dcd >2014-08-21 11:42:17,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:17,247 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:17,278 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:17,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4ddb8ca2 >2014-08-21 11:42:17,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, FenceVdsVDSCommand, return: Done, log id: 4ddb8ca2 >2014-08-21 11:42:17,403 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:42:22,404 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:42:22,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,500 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:22,532 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) [7834da4e] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:22,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4a2e75a0 >2014-08-21 11:42:22,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) [7834da4e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 4a2e75a0 >2014-08-21 11:42:22,662 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-3) [7834da4e] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:42:22,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7834da4e] Correlation ID: 7834da4e, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:42:22,708 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-3) [55a33e7d] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:22,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,843 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:22,848 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:42:22,882 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 237fbb5d >2014-08-21 11:42:22,886 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, SetVdsStatusVDSCommand, log id: 237fbb5d >2014-08-21 11:42:22,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:22,980 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:23,011 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:23,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 420a63f7 >2014-08-21 11:42:23,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Done, log id: 420a63f7 >2014-08-21 11:42:23,134 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:42:28,135 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:42:28,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:42:28,204 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:42:28,235 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-3) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:42:28,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4333599a >2014-08-21 11:42:28,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-3) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4333599a >2014-08-21 11:42:28,366 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-3) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:42:28,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: 4ba29363, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:42:28,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_syncAction_b264b449-6c7a-417a, Job ID: cdf4d938-6490-4866-959f-ab365bd879d1, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:42:28,377 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-3) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:44:06,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:44:06,238 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:44:06,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:44:09,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 788c62eb >2014-08-21 11:44:09,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetHardwareInfoVDSCommand, log id: 788c62eb >2014-08-21 11:44:09,921 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [21c808bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:44:09,932 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-49) [f746b40] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:44:09,938 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:44:09,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2914d44e >2014-08-21 11:44:10,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 2914d44e >2014-08-21 11:44:10,222 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:44:10,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6528dcdb >2014-08-21 11:44:10,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6528dcdb >2014-08-21 11:44:10,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:44:10,640 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:44:10,682 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:44:10,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 113f7e16 >2014-08-21 11:44:11,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 113f7e16 >2014-08-21 11:44:11,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 3e4865c4 >2014-08-21 11:44:11,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3e4865c4 >2014-08-21 11:44:11,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:44:11,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7c0f36a0] Correlation ID: f746b40, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:44:11,495 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [54395203] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:28,803 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:45:29,970 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,971 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,972 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-3) [hosts_update_563fb608-6a6c-4c9e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:29,976 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:29,976 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-3) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,003 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:45:30,010 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: hosts_update_563fb608-6a6c-4c9e, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:45:30,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,250 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,251 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [hosts_update_d6a998f1-f386-4c34] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:45:30,285 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,286 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:45:30,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:45:30,486 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-2) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:45:30,517 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-2) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:45:30,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-2) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4dedd0b >2014-08-21 11:45:30,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-2) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4dedd0b >2014-08-21 11:45:30,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: hosts_update_d6a998f1-f386-4c34, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:45:59,658 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 in problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:47:21,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server monique-vds01.tlv.redhat.com failed to respond >2014-08-21 11:47:21,207 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:21,209 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-44) [697a661d] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 11:47:21,248 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [697a661d] Correlation ID: 697a661d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-21 11:47:21,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-08-21 11:47:21,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-76) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 11:47:21,278 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:47:21,279 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:47:21,331 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-21 11:47:21,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-21 11:47:21,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-21 11:47:21,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:47:21,401 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 11:47:21,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 11:47:21,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 18f2a9a1 >2014-08-21 11:47:21,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] SpmStopVDSCommand:: vds monique-vds01.tlv.redhat.com is in Connecting status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:47:21,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] FINISH, SpmStopVDSCommand, log id: 18f2a9a1 >2014-08-21 11:47:21,408 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [697a661d] spm stop on spm failed, stopping spm selection! >2014-08-21 11:47:24,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3d1203f9 >2014-08-21 11:47:24,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:47:24,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command GetCapabilitiesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-21 11:47:24,430 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is initializing. Message: Recovering from crash or Initializing >2014-08-21 11:47:24,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Failed to refresh VDS , vds = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2014-08-21 11:47:27,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: fbbec82 >2014-08-21 11:47:27,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GetHardwareInfoVDSCommand, log id: fbbec82 >2014-08-21 11:47:27,954 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [e9cdbb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:47:28,000 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [e9cdbb4] Cancelling the recovery from crash timer for VDS 9e129c9c-c9d3-42a5-a28e-b0c47c89affb because vds started initializing >2014-08-21 11:47:28,002 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [25f17de4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:28,008 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [43630027] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:47:28,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [43630027] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bfb965 >2014-08-21 11:47:31,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:47:31,505 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM Init: could not find reported vds or not up - pool:DC1 vds_spm_id: 1 >2014-08-21 11:47:31,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) SPM selection - vds seems as spm monique-vds01.tlv.redhat.com >2014-08-21 11:47:31,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-97) START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 6a5634c5 >2014-08-21 11:47:31,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-08-21 11:47:31,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:47:31,589 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-08-21 11:47:31,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, SpmStopVDSCommand, log id: 6a5634c5 >2014-08-21 11:47:31,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) spm stop succeeded, continuing with spm selection >2014-08-21 11:47:31,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId 1, LVER 7 >2014-08-21 11:47:31,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=1, prevLVER=7, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 393a12c1 >2014-08-21 11:47:31,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling started: taskId = ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4 >2014-08-21 11:47:44,765 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:47:55,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling ended: taskId = ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4 task status = finished >2014-08-21 11:47:55,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) spmStart polling ended, spm status: SPM >2014-08-21 11:47:55,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=ea1a8e38-2cbe-452a-b65e-30c6ebc9dcb4), log id: 63b593aa >2014-08-21 11:47:55,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, HSMClearTaskVDSCommand, log id: 63b593aa >2014-08-21 11:47:55,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f6cb33c, log id: 393a12c1 >2014-08-21 11:47:55,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 11:47:55,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:47:55,287 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:47:55,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 446f1df1 >2014-08-21 11:47:55,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:47:55,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 69b57d3c >2014-08-21 11:47:55,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69b57d3c >2014-08-21 11:47:55,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 446f1df1 >2014-08-21 11:47:55,384 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-35) Discovered no tasks on Storage Pool DC1 >2014-08-21 11:48:06,877 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-08-21 11:48:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-08-21 11:48:28,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [43630027] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 4bfb965 >2014-08-21 11:48:28,086 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [43630027] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 11:48:28,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6e9fdc0f >2014-08-21 11:49:28,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a']] >2014-08-21 11:49:28,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:49:28,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) Command ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a' >2014-08-21 11:49:28,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 6e9fdc0f >2014-08-21 11:49:28,537 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-12) Could not connect host monique-vds01.tlv.redhat.com to pool DC1 with the message: null >2014-08-21 11:49:28,724 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:49:28,727 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 10c185 >2014-08-21 11:49:28,844 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [e28c63d] FINISH, SetVdsStatusVDSCommand, log id: 10c185 >2014-08-21 11:49:28,996 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: e28c63d, Job ID: a469e856-d59f-4b83-8715-08b983d77cf3, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-08-21 11:49:29,109 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 11:49:29,128 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [e28c63d] Correlation ID: 25f17de4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host monique-vds01.tlv.redhat.com to Storage Pool DC1 >2014-08-21 11:49:29,141 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [5806b578] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:49:29,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [5806b578] Host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb : monique-vds01.tlv.redhat.com is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-08-21 11:49:40,128 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) [hosts_syncAction_35ae1321-07b3-41f2] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:49:40,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:40,336 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:40,341 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: RestartVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:40,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:40,472 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:40,503 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:40,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d0253f6 >2014-08-21 11:49:41,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3d0253f6 >2014-08-21 11:49:41,161 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:41,192 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 45e405f1 >2014-08-21 11:49:41,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 11:49:41,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 15237350 >2014-08-21 11:49:41,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 527aa183 >2014-08-21 11:49:41,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] SpmStopVDSCommand::Stopping SPM on vds rose07.qa.lab.tlv.redhat.com, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:49:41,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, SpmStopVDSCommand, log id: 527aa183 >2014-08-21 11:49:41,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, ResetIrsVDSCommand, log id: 15237350 >2014-08-21 11:49:41,288 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, SetVdsStatusVDSCommand, log id: 45e405f1 >2014-08-21 11:49:41,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:41,420 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:41,451 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:41,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d7db5ed >2014-08-21 11:49:41,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, FenceVdsVDSCommand, return: Done, log id: 3d7db5ed >2014-08-21 11:49:41,576 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 11:49:46,108 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : no reporting hosts >2014-08-21 11:49:46,113 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [22a988a8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:49:46,119 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [22a988a8] Correlation ID: 22a988a8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:49:46,577 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:49:46,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:46,674 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:46,706 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) [1ef77c35] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:46,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 6bc37cdf >2014-08-21 11:49:46,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 6bc37cdf >2014-08-21 11:49:46,837 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-7) [1ef77c35] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 11:49:46,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1ef77c35] Correlation ID: 1ef77c35, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-08-21 11:49:46,883 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp--127.0.0.1-8702-7) [4fbd435b] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:47,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:47,073 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:47,077 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:49:47,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5de3ab47 >2014-08-21 11:49:47,143 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVdsStatusVDSCommand, log id: 5de3ab47 >2014-08-21 11:49:47,272 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:47,273 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:47,352 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:47,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4b6d160a >2014-08-21 11:49:47,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Done, log id: 4b6d160a >2014-08-21 11:49:47,475 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 11:49:52,476 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 11:49:52,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:49:52,625 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:49:52,662 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:49:52,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 3d07d294 >2014-08-21 11:49:52,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3d07d294 >2014-08-21 11:49:52,808 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (ajp--127.0.0.1-8702-7) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 11:49:52,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 6a8e1df6, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-08-21 11:49:52,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: hosts_syncAction_35ae1321-07b3-41f2, Job ID: 888a0e54-6915-4c04-877a-c3b41ce44c17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-08-21 11:49:52,820 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp--127.0.0.1-8702-7) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 11:50:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering 1 hosts >2014-08-21 11:50:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering hosts id: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, name : monique-vds01.tlv.redhat.com >2014-08-21 11:50:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock Acquired to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS >, sharedLocks= ] >2014-08-21 11:50:00,017 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:50:00,018 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Before acquiring lock in order to prevent monitoring for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,019 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock acquired, from now a monitoring of host will be skipped for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: dcf92b1 >2014-08-21 11:50:00,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] FINISH, SetVdsStatusVDSCommand, log id: dcf92b1 >2014-08-21 11:50:00,093 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Activate finished. Lock released. Monitoring can run now for host monique-vds01.tlv.redhat.com from data-center DC1 >2014-08-21 11:50:00,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Correlation ID: 53b2f72b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was autorecovered. >2014-08-21 11:50:00,098 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Lock freed to object EngineLock [exclusiveLocks= key: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb value: VDS >, sharedLocks= ] >2014-08-21 11:50:02,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vds=Host[monique-vds01.tlv.redhat.com,9e129c9c-c9d3-42a5-a28e-b0c47c89affb]), log id: 78d23540 >2014-08-21 11:50:02,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 78d23540 >2014-08-21 11:50:02,334 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [78f9a316] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:50:02,382 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [5edecdb3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:50:02,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:50:02,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5bf512c1 >2014-08-21 11:50:59,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 >2014-08-21 11:50:59,750 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-21 11:50:59,779 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [2b61e701] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-21 11:50:59,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [2b61e701] Removing vds [] from the domain in maintenance cache >2014-08-21 11:50:59,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [2b61e701] Removing host(s) [] from hosts unseen domain report cache >2014-08-21 11:50:59,800 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) Finished reconstruct for pool ab2440ae-4286-4465-beed-c2ccc8256f8b. Clearing event queue >2014-08-21 11:51:02,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 5bf512c1 >2014-08-21 11:51:02,449 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-08-21 11:51:02,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: c8b92d8 >2014-08-21 11:51:32,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 11:51:32,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-21 11:51:32,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:51:36,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 2d562b1a >2014-08-21 11:51:36,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 2d562b1a >2014-08-21 11:51:36,274 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [39c10079] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:51:36,283 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [6adcedc2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:51:36,289 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:51:36,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74522469 >2014-08-21 11:51:36,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 74522469 >2014-08-21 11:51:36,542 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 11:52:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a']] >2014-08-21 11:52:02,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) HostName = monique-vds01.tlv.redhat.com >2014-08-21 11:52:02,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) Command ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ab2440ae-4286-4465-beed-c2ccc8256f8b, msdUUID=59645b6f-d70b-45fb-aa66-c582c373d58a' >2014-08-21 11:52:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, ConnectStoragePoolVDSCommand, log id: c8b92d8 >2014-08-21 11:52:02,871 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-34) Could not connect host monique-vds01.tlv.redhat.com to pool DC1, as the master domain is in inactive/unknown status - not failing the operation >2014-08-21 11:52:02,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 45009897 >2014-08-21 11:52:02,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 11a8fb51 >2014-08-21 11:52:03,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [71755e69] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45009897 >2014-08-21 11:52:03,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [71755e69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-08-21 11:52:03,033 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [71755e69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-08-21 11:52:03,044 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [336ed24f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 9e129c9c-c9d3-42a5-a28e-b0c47c89affb Type: VDS >2014-08-21 11:52:03,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 11a8fb51 >2014-08-21 11:52:03,156 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 in problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:52:03,252 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:52:03,253 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [5d052fe] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 11:52:03,285 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-9) [5d052fe] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:secure=false >2014-08-21 11:52:03,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = 'secure=false'), log id: 4e499d08 >2014-08-21 11:52:03,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 4e499d08 >2014-08-21 11:52:03,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 4ce51c58 >2014-08-21 11:52:03,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [5d052fe] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4ce51c58 >2014-08-21 11:52:03,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 11:52:03,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [5d052fe] Correlation ID: 6adcedc2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 11:52:03,614 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [656ae0c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:52:06,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-08-21 11:52:06,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, vdsId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 55bfa7be >2014-08-21 11:52:53,517 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource] (ajp--127.0.0.1-8702-2) Unwrapping of 'from' matrix search parameter failed.: java.lang.NumberFormatException: For input string: "" > at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:453) [rt.jar:1.7.0_51] > at java.lang.Long.parseLong(Long.java:483) [rt.jar:1.7.0_51] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getSearchParameters(AbstractBackendCollectionResource.java:72) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:55) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:51) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.getBackendCollection(BackendEventsResource.java:57) [restapi-jaxrs.jar:] > at org.ovirt.engine.api.restapi.resource.BackendEventsResource.list(BackendEventsResource.java:31) [restapi-jaxrs.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 11:52:56,493 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,494 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,495 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [hosts_update_37ba5b3f-df29-4084] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 11:52:56,501 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,502 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 11:52:56,530 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 11:52:56,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: hosts_update_37ba5b3f-df29-4084, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 11:52:57,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, ConnectStoragePoolVDSCommand, log id: 55bfa7be >2014-08-21 11:52:57,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] hostFromVds::Connected host to pool - selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:52:57,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [235d3283] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:52:57,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6eee7651 >2014-08-21 11:52:57,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling started: taskId = d8128341-9a52-4b5b-90e8-5238c9c61cd9 >2014-08-21 11:52:58,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Failed in HSMGetTaskStatusVDS method >2014-08-21 11:52:58,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling ended: taskId = d8128341-9a52-4b5b-90e8-5238c9c61cd9 task status = finished >2014-08-21 11:52:58,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id, code = 661 >2014-08-21 11:52:59,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] spmStart polling ended, spm status: Free >2014-08-21 11:52:59,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=d8128341-9a52-4b5b-90e8-5238c9c61cd9), log id: 50fe5792 >2014-08-21 11:52:59,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, HSMClearTaskVDSCommand, log id: 50fe5792 >2014-08-21 11:52:59,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [235d3283] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@593504fc, log id: 6eee7651 >2014-08-21 11:52:59,087 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 11:52:59,095 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [27da6a25] Correlation ID: 27da6a25, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-08-21 11:52:59,099 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-08-21 11:52:59,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] Irs placed on server 9e129c9c-c9d3-42a5-a28e-b0c47c89affb failed. Proceed Failover >2014-08-21 11:52:59,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 11:52:59,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-08-21 11:52:59,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 794c3b34 >2014-08-21 11:52:59,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling started: taskId = 76a348e2-0af2-456f-b08d-360d146e84d9 >2014-08-21 11:53:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling ended: taskId = 76a348e2-0af2-456f-b08d-360d146e84d9 task status = finished >2014-08-21 11:53:00,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] spmStart polling ended, spm status: SPM >2014-08-21 11:53:00,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, taskId=76a348e2-0af2-456f-b08d-360d146e84d9), log id: 626a4429 >2014-08-21 11:53:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] FINISH, HSMClearTaskVDSCommand, log id: 626a4429 >2014-08-21 11:53:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [27da6a25] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48c1753b, log id: 794c3b34 >2014-08-21 11:53:00,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [27da6a25] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-08-21 11:53:00,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [27da6a25] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-08-21 11:53:00,427 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 11:53:00,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 3dc4fde6 >2014-08-21 11:53:00,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 11:53:00,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 1567bc3c >2014-08-21 11:53:00,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1567bc3c >2014-08-21 11:53:00,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) [27da6a25] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dc4fde6 >2014-08-21 11:53:00,588 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-2) [27da6a25] Discovered no tasks on Storage Pool DC1 >2014-08-21 11:53:03,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 recovered from problem. vds: monique-vds01.tlv.redhat.com >2014-08-21 11:53:03,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 59645b6f-d70b-45fb-aa66-c582c373d58a:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-21 12:09:41,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:09:41,897 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:09:41,900 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 12:09:41,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 2b65d4c7 >2014-08-21 12:09:42,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2b65d4c7 >2014-08-21 12:09:48,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:09:48,684 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:09:48,687 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:blablabla.com, User:asfasf, Options: >2014-08-21 12:09:48,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blablabla.com, port = , type = ipmilan, user = asfasf, password = ******, options = ''), log id: 5d2bbda5 >2014-08-21 12:10:27,346 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed] >2014-08-21 12:10:27,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed], log id: 5d2bbda5 >2014-08-21 12:10:27,349 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Fencing operation failed with proxy host 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, trying another proxy... >2014-08-21 12:10:27,360 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to run Power Management command on Host , no running proxy Host was found. >2014-08-21 12:10:27,361 WARN [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-08-21 12:10:27,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:10:27,372 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 12:10:27,373 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:blablabla.com, User:asfasf, Options: >2014-08-21 12:10:27,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = blablabla.com, port = , type = ipmilan, user = asfasf, password = ******, options = ''), log id: 28301e85 >2014-08-21 12:11:05,589 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.[Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed] >2014-08-21 12:11:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Failed, [Getting status of IPMI:blablabla.com...Chassis power = Unknown, Failed], log id: 28301e85 >2014-08-21 12:11:31,369 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,370 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,371 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-1) [7560924b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:11:31,375 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,376 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 12:11:31,435 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:11:31,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: 7560924b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 12:20:28,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-08-21 12:20:28,996 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 12:20:28,997 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_51] > at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_51] > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) [jsse.jar:1.7.0_51] > at sun.security.ssl.InputRecord.read(InputRecord.java:480) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) [jsse.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) [jsse.jar:1.7.0_51] > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) [jsse.jar:1.7.0_51] > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) [rt.jar:1.7.0_51] > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) [rt.jar:1.7.0_51] > at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 12:20:29,029 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [53b2f72b] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-08-21 12:20:32,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 42b9f1bf >2014-08-21 12:20:32,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GetHardwareInfoVDSCommand, log id: 42b9f1bf >2014-08-21 12:20:32,133 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [70f9c04d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:20:32,183 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [64dc3cae] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 12:20:32,188 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 12:20:32,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20ad6e18 >2014-08-21 12:20:32,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 20ad6e18 >2014-08-21 12:20:32,245 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 12:20:32,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 60c6b02e >2014-08-21 12:20:32,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 60c6b02e >2014-08-21 12:20:32,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 196aebf7 >2014-08-21 12:20:32,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [fbb240a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 196aebf7 >2014-08-21 12:20:32,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [fbb240a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 12:20:32,613 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [fbb240a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 12:20:32,630 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [174c1abd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 12:48:06,885 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-08-21 12:48:07,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-08-21 13:48:06,923 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-08-21 13:48:07,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-08-21 14:47:32,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:47:32,636 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host >2014-08-21 14:47:32,638 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-7) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:47:32,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 7e5772fe >2014-08-21 14:47:33,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7e5772fe >2014-08-21 14:47:34,783 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,784 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,785 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-4) [4d400bc4] Running command: UpdateVdsCommand internal: false. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:47:34,806 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:34,807 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-4) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-21 14:47:35,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:47:35,010 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:47:35,041 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:47:35,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 57d71205 >2014-08-21 14:47:35,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 57d71205 >2014-08-21 14:47:35,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 4d400bc4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-08-21 14:48:06,927 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-08-21 14:48:07,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-08-21 14:51:04,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [22a988a8] Command SpmStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 14:51:04,147 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:51:04,149 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Storage Pool ab2440ae-4286-4465-beed-c2ccc8256f8b - Updating Storage Domain 59645b6f-d70b-45fb-aa66-c582c373d58a status from Active to Unknown, reason : null >2014-08-21 14:51:04,187 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Correlation ID: 1679f6b2, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host rose07.qa.lab.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-08-21 14:51:04,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-86) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-08-21 14:51:04,262 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:04,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.util.concurrent.TimeoutException > at java.util.concurrent.FutureTask.get(FutureTask.java:201) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy.invoke(XmlRpcUtils.java:200) [vdsbroker.jar:] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsServerWrapper.list(VdsServerWrapper.java:127) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:23) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > ... 14 more > >2014-08-21 14:51:04,276 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.util.concurrent.TimeoutException >2014-08-21 14:51:09,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) [799af561] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:09,440 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:09,441 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [799af561] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:09,460 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) [799af561] Failed to refresh VDS , vds = a8fdd580-c752-4537-9cae-e49e468db7da : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,500 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:14,554 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Server failed to respond, vds_id = a8fdd580-c752-4537-9cae-e49e468db7da, vds_name = rose07.qa.lab.tlv.redhat.com, vm_count = 0, spm_status = SPM, non-responsive_timeout (seconds) = 80, error = java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:14,592 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is non responsive. >2014-08-21 14:51:14,606 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-1) ResourceManager::vdsNotResponding entered for Host a8fdd580-c752-4537-9cae-e49e468db7da, rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:14,617 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:14,671 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:51:14,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:51:14,800 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:14,830 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:51:14,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 3b5347cf >2014-08-21 14:51:14,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 3b5347cf >2014-08-21 14:51:14,972 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] Opening SSH Soft Fencing session on host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:51:19,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:19,688 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:19,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:24,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) [697a661d] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:24,750 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) [697a661d] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:24,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [697a661d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:29,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:29,811 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:29,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:34,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:34,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:34,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:39,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:39,941 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:39,946 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:45,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:45,012 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:45,017 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:50,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:50,073 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:50,078 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:51:55,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:51:55,135 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:51:55,140 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:00,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:00,196 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:00,200 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [aa7b6a4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:05,255 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [6102b316] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:05,256 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [6102b316] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:05,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [6102b316] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:10,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) [36ee3909] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:10,318 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) [36ee3909] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:10,323 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [36ee3909] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:15,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:15,379 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:15,385 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:17,990 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-1) [2ff739c2] SSH Soft Fencing command failed on host rose07.qa.lab.tlv.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1135) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1254) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1874) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1275) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:345) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$3.run(VdsEventListener.java:221) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_51] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_51] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_51] > ... 1 more > >2014-08-21 14:52:18,109 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:18,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:18,214 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Restart command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:18,218 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:18,295 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:20,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:20,448 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:20,454 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:25,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:25,510 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:25,515 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:30,570 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [336ed24f] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:30,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [336ed24f] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:30,577 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [336ed24f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:35,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:35,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) [3a17e524] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:35,637 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [3a17e524] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:40,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:40,692 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:40,698 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:45,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:45,754 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:45,789 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:48,339 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2453023d] Correlation ID: 2453023d, Job ID: 0dc11331-e4f5-49df-8cd6-bc32aea78fb3, Call Stack: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host 'rose07.qa.lab.tlv.redhat.com'. >2014-08-21 14:52:48,343 INFO [org.ovirt.engine.core.bll.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:48,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [2453023d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:48,442 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2453023d] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:48,473 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [2453023d] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:48,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: cf1678b >2014-08-21 14:52:49,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2453023d] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: cf1678b >2014-08-21 14:52:49,122 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Lock Acquired to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:52:49,127 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Running command: StopVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:49,159 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12552137 >2014-08-21 14:52:50,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [5806b578] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:50,849 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [5806b578] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:50,854 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [5806b578] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:52:50,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-08-21 14:52:50,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 113897f >2014-08-21 14:52:50,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 284e18bb >2014-08-21 14:52:50,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] SpmStopVDSCommand:: vds rose07.qa.lab.tlv.redhat.com is in NonResponsive status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:52:50,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SpmStopVDSCommand, log id: 284e18bb >2014-08-21 14:52:50,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, ResetIrsVDSCommand, log id: 113897f >2014-08-21 14:52:50,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SetVdsStatusVDSCommand, log id: 12552137 >2014-08-21 14:52:51,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:51,014 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:51,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b), log id: 634426cc >2014-08-21 14:52:51,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] SpmStopVDSCommand:: vds rose07.qa.lab.tlv.redhat.com is in Reboot status - not performing spm stop, pool id ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:52:51,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, SpmStopVDSCommand, log id: 634426cc >2014-08-21 14:52:51,116 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Executing <Stop> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:51,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 16f4002e >2014-08-21 14:52:51,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, FenceVdsVDSCommand, return: Done, log id: 16f4002e >2014-08-21 14:52:51,271 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-08-21 14:52:56,272 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 14:52:56,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,338 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,369 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:56,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 33ad64a2 >2014-08-21 14:52:56,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 33ad64a2 >2014-08-21 14:52:56,504 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-8-thread-1) [6f660a90] vds rose07.qa.lab.tlv.redhat.com status is off >2014-08-21 14:52:56,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6f660a90] Correlation ID: 6f660a90, Job ID: 79a42b3e-8ff2-4840-9acf-ace2559d63fc, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by Engine. >2014-08-21 14:52:56,586 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:56,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] START, ResetIrsVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, ignoreStopFailed = false), log id: 629d090 >2014-08-21 14:52:56,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: SocketException: Socket closed >2014-08-21 14:52:56,610 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [1679f6b2] IRS failover failed - cant allocate vds server >2014-08-21 14:52:56,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [64995578] FINISH, ResetIrsVDSCommand, log id: 629d090 >2014-08-21 14:52:56,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,757 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Running command: StartVdsCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:52:56,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 72f871d4 >2014-08-21 14:52:56,794 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, SetVdsStatusVDSCommand, log id: 72f871d4 >2014-08-21 14:52:56,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:52:56,893 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:52:56,940 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Executing <Start> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:52:56,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 4908bf9c >2014-08-21 14:52:57,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, FenceVdsVDSCommand, return: Done, log id: 4908bf9c >2014-08-21 14:52:57,127 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-08-21 14:52:58,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:52:58,976 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:52:58,981 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:02,127 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-08-21 14:53:02,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:53:02,193 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:53:02,223 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-1) Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:53:02,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 2b096ecb >2014-08-21 14:53:02,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2b096ecb >2014-08-21 14:53:02,357 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-8-thread-1) vds rose07.qa.lab.tlv.redhat.com status is on >2014-08-21 14:53:02,360 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: 505f5d3c, Job ID: 3757189b-195f-459b-a1f7-cbb5fdd79b36, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by Engine. >2014-08-21 14:53:02,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) Correlation ID: 2453023d, Job ID: fc223561-9382-4d84-9c98-8dae3ae70120, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is rebooting. >2014-08-21 14:53:02,369 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-1) Lock freed to object EngineLock [exclusiveLocks= key: a8fdd580-c752-4537-9cae-e49e468db7da value: VDS_FENCE >, sharedLocks= ] >2014-08-21 14:53:04,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:04,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:04,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:06,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-08-21 14:53:06,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId 2, LVER 9 >2014-08-21 14:53:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, prevId=2, prevLVER=9, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20a545ba >2014-08-21 14:53:06,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling started: taskId = def043da-6eb2-4b6a-8516-bcb5c0572940 >2014-08-21 14:53:07,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling ended: taskId = def043da-6eb2-4b6a-8516-bcb5c0572940 task status = finished >2014-08-21 14:53:07,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] spmStart polling ended, spm status: SPM >2014-08-21 14:53:07,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, taskId=def043da-6eb2-4b6a-8516-bcb5c0572940), log id: 1990638c >2014-08-21 14:53:07,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, HSMClearTaskVDSCommand, log id: 1990638c >2014-08-21 14:53:07,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-80) [3a17e524] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63fb86f5, log id: 20a545ba >2014-08-21 14:53:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [3a17e524] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-08-21 14:53:07,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [3a17e524] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-08-21 14:53:07,823 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-21 14:53:07,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, ignoreFailoverLimit = false), log id: 4a97a40 >2014-08-21 14:53:07,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] -- executeIrsBrokerCommand: Attempting on storage pool ab2440ae-4286-4465-beed-c2ccc8256f8b >2014-08-21 14:53:07,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb), log id: 2b482353 >2014-08-21 14:53:07,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b482353 >2014-08-21 14:53:07,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) [3a17e524] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a97a40 >2014-08-21 14:53:07,908 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) [3a17e524] Discovered no tasks on Storage Pool DC1 >2014-08-21 14:53:09,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:09,113 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:09,118 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:14,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:14,197 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:14,202 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:19,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:19,258 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:19,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:24,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:24,323 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:24,335 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:29,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:29,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:29,415 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:34,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [e83a83] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:34,473 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [e83a83] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:34,478 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [e83a83] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:39,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:39,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:39,542 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:44,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:44,617 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:44,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:49,675 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:49,676 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:49,681 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:54,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:54,738 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:54,743 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:53:59,795 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:53:59,797 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:53:59,801 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [1679f6b2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:04,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:04,858 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:04,868 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:09,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:09,925 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:09,930 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:15,007 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) [351eedf6] Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:15,009 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) [351eedf6] Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:15,015 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) [351eedf6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:20,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:20,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:20,077 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:25,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:25,133 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:25,138 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:30,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-08-21 14:54:30,194 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of $120 seconds and after that an attempt to fence the host will be issued. >2014-08-21 14:54:30,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_51] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-08-21 14:54:34,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vds=Host[rose07.qa.lab.tlv.redhat.com,a8fdd580-c752-4537-9cae-e49e468db7da]), log id: 47ab6863 >2014-08-21 14:54:34,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GetHardwareInfoVDSCommand, log id: 47ab6863 >2014-08-21 14:54:34,877 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [499d76ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS >2014-08-21 14:54:34,886 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [385cd29d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:54:34,891 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ab2440ae-4286-4465-beed-c2ccc8256f8b Type: StoragePool >2014-08-21 14:54:34,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, storageType = NFS, connectionList = [{ id: ca0e9d8b-099d-418a-832f-2a177a5e92ed, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs_3.5, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6057c658 >2014-08-21 14:54:35,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, ConnectStorageServerVDSCommand, return: {ca0e9d8b-099d-418a-832f-2a177a5e92ed=0}, log id: 6057c658 >2014-08-21 14:54:35,161 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-08-21 14:54:35,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da, vdsId = a8fdd580-c752-4537-9cae-e49e468db7da, storagePoolId = ab2440ae-4286-4465-beed-c2ccc8256f8b, masterVersion = 1), log id: 6e8e619f >2014-08-21 14:54:35,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 6e8e619f >2014-08-21 14:54:35,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-08-21 14:54:35,511 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) [d7111e2] Using Host monique-vds01.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-08-21 14:54:35,542 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-50) [d7111e2] Executing <Status> Power Management command, Proxy Host:monique-vds01.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options: >2014-08-21 14:54:35,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, FenceVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 9e129c9c-c9d3-42a5-a28e-b0c47c89affb, targetVdsId = a8fdd580-c752-4537-9cae-e49e468db7da, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = ''), log id: 23888b6d >2014-08-21 14:54:35,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 23888b6d >2014-08-21 14:54:35,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = a8fdd580-c752-4537-9cae-e49e468db7da), log id: 38a8537e >2014-08-21 14:54:35,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-50) [d7111e2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 38a8537e >2014-08-21 14:54:35,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-08-21 14:54:35,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [d7111e2] Correlation ID: 385cd29d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-08-21 14:54:35,893 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [22fbf874] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a8fdd580-c752-4537-9cae-e49e468db7da Type: VDS
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1070674
: 929190