Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 949888 Details for
Bug 1156055
[external-task] Couldn't clear external job
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
23_10_14_engine.log (text/plain), 1.38 MB, created by
sefi litmanovich
on 2014-10-23 13:39:36 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
sefi litmanovich
Created:
2014-10-23 13:39:36 UTC
Size:
1.38 MB
patch
obsolete
>2014-10-15 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) [30c3d974] Start deleteAgedOutAuditLogs >2014-10-15 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-94) Start deleteAgedOutCommandEntities >2014-10-15 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) [30c3d974] Finished deleteAgedOutAuditLogs >2014-10-15 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-94) Finished deleteAgedOutCommandEntities >2014-10-15 03:43:49,488 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-10-15 04:43:49,506 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [10ec4770] Attempting to update VMs/Templates Ovf. >2014-10-15 05:43:49,521 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-10-15 06:43:49,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-10-15 07:43:49,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-10-15 08:43:49,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-15 09:43:49,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-10-15 10:43:49,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-10-15 11:43:49,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-15 12:43:49,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-10-15 13:43:49,685 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) [6daba33e] Attempting to update VMs/Templates Ovf. >2014-10-15 14:43:49,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-10-15 15:43:49,716 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-10-15 16:43:49,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-10-15 17:43:49,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) [7d5570eb] Attempting to update VMs/Templates Ovf. >2014-10-15 18:43:49,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-10-15 19:43:49,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-10-15 20:43:49,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-10-15 21:43:49,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-10-15 22:43:49,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-10-15 23:43:49,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-16 00:43:49,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [a6d9186] Attempting to update VMs/Templates Ovf. >2014-10-16 01:43:49,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [701ca55e] Attempting to update VMs/Templates Ovf. >2014-10-16 02:43:49,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-10-16 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Start deleteAgedOutCommandEntities >2014-10-16 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Start deleteAgedOutAuditLogs >2014-10-16 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Finished deleteAgedOutAuditLogs >2014-10-16 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Finished deleteAgedOutCommandEntities >2014-10-16 03:43:49,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-10-16 04:43:49,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-10-16 05:43:49,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-10-16 06:43:49,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-10-16 07:43:50,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-10-16 08:43:50,018 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-10-16 09:43:50,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-10-16 10:43:50,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [3e41b5f3] Attempting to update VMs/Templates Ovf. >2014-10-16 11:43:50,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-10-16 12:43:50,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [179cd8b8] Attempting to update VMs/Templates Ovf. >2014-10-16 13:43:50,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-10-16 14:43:50,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-10-16 15:43:50,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-10-16 16:43:50,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-10-16 17:43:50,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) [2d07f4c3] Attempting to update VMs/Templates Ovf. >2014-10-16 18:43:50,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) [78c9ce85] Attempting to update VMs/Templates Ovf. >2014-10-16 19:43:50,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-10-16 20:43:50,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-10-16 21:43:50,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-10-16 22:43:50,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-10-16 23:43:50,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-10-17 00:43:50,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) [6daba33e] Attempting to update VMs/Templates Ovf. >2014-10-17 01:43:50,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-10-17 02:43:50,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [10ec4770] Attempting to update VMs/Templates Ovf. >2014-10-17 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Start deleteAgedOutCommandEntities >2014-10-17 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-39) Start deleteAgedOutAuditLogs >2014-10-17 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-39) Finished deleteAgedOutAuditLogs >2014-10-17 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Finished deleteAgedOutCommandEntities >2014-10-17 03:43:50,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [179cd8b8] Attempting to update VMs/Templates Ovf. >2014-10-17 04:43:50,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-10-17 05:43:50,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-10-17 06:43:50,411 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [534fb9eb] Attempting to update VMs/Templates Ovf. >2014-10-17 07:43:50,428 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-10-17 08:43:50,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [5916475a] Attempting to update VMs/Templates Ovf. >2014-10-17 09:43:50,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-10-17 10:43:50,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [a6d9186] Attempting to update VMs/Templates Ovf. >2014-10-17 11:43:50,496 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-10-17 12:43:50,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) [50ac9031] Attempting to update VMs/Templates Ovf. >2014-10-17 13:43:50,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-10-17 14:43:50,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-10-17 15:43:50,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [30c3d974] Attempting to update VMs/Templates Ovf. >2014-10-17 16:43:50,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-10-17 17:43:50,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) [4d483c7a] Attempting to update VMs/Templates Ovf. >2014-10-17 18:43:50,627 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-10-17 19:43:50,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-10-17 20:43:50,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-10-17 21:43:50,679 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) [50ac9031] Attempting to update VMs/Templates Ovf. >2014-10-17 22:43:50,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) [7d5570eb] Attempting to update VMs/Templates Ovf. >2014-10-17 23:43:50,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-10-18 00:43:50,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) [4d483c7a] Attempting to update VMs/Templates Ovf. >2014-10-18 01:43:50,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-10-18 02:43:50,759 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-10-18 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Start deleteAgedOutAuditLogs >2014-10-18 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-51) Start deleteAgedOutCommandEntities >2014-10-18 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Finished deleteAgedOutAuditLogs >2014-10-18 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-51) Finished deleteAgedOutCommandEntities >2014-10-18 03:43:50,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [a6d9186] Attempting to update VMs/Templates Ovf. >2014-10-18 04:43:50,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-10-18 05:43:50,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-10-18 06:43:50,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-10-18 07:43:50,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-10-18 08:43:50,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-10-18 09:43:50,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-10-18 10:43:50,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-10-18 11:43:50,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-10-18 12:43:50,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-10-18 13:43:50,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-18 14:43:50,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-10-18 15:43:51,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-10-18 16:43:51,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-10-18 17:43:51,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-10-18 18:43:51,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) [78c9ce85] Attempting to update VMs/Templates Ovf. >2014-10-18 19:43:51,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-10-18 20:43:51,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-10-18 21:43:51,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-10-18 22:43:51,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-10-18 23:43:51,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-10-19 00:43:51,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-10-19 01:43:51,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [534fb9eb] Attempting to update VMs/Templates Ovf. >2014-10-19 02:43:51,246 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-10-19 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) [78c9ce85] Start deleteAgedOutCommandEntities >2014-10-19 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) [3582a930] Start deleteAgedOutAuditLogs >2014-10-19 03:35:35,012 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) [3582a930] Finished deleteAgedOutAuditLogs >2014-10-19 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-59) [78c9ce85] Finished deleteAgedOutCommandEntities >2014-10-19 03:43:51,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [a6d9186] Attempting to update VMs/Templates Ovf. >2014-10-19 04:43:51,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-10-19 05:43:51,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-10-19 06:43:51,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-10-19 07:43:51,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-10-19 08:43:51,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) [7d5570eb] Attempting to update VMs/Templates Ovf. >2014-10-19 09:43:51,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [4d2ae53d] Attempting to update VMs/Templates Ovf. >2014-10-19 10:43:51,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [534fb9eb] Attempting to update VMs/Templates Ovf. >2014-10-19 11:43:51,413 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-19 12:43:51,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-10-19 13:43:51,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-10-19 14:43:51,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-10-19 15:43:51,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-10-19 16:43:51,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-10-19 17:43:51,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-10-19 18:43:51,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-10-19 19:43:51,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [3e41b5f3] Attempting to update VMs/Templates Ovf. >2014-10-19 20:43:51,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-10-19 21:43:51,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [25e8119] Attempting to update VMs/Templates Ovf. >2014-10-19 22:43:51,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-10-19 23:43:51,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [10ec4770] Attempting to update VMs/Templates Ovf. >2014-10-20 00:43:51,658 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [534fb9eb] Attempting to update VMs/Templates Ovf. >2014-10-20 01:43:51,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) [15c82c98] Attempting to update VMs/Templates Ovf. >2014-10-20 02:43:51,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-10-20 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) [3e41b5f3] Start deleteAgedOutAuditLogs >2014-10-20 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Start deleteAgedOutCommandEntities >2014-10-20 03:35:35,005 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) [3e41b5f3] Finished deleteAgedOutAuditLogs >2014-10-20 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Finished deleteAgedOutCommandEntities >2014-10-20 03:43:51,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-10-20 04:43:51,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-10-20 04:48:23,066 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-40) domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 in problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 04:48:38,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-5) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 recovered from problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 04:48:38,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-5) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-10-20 04:48:57,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [15c82c98] Failed in GetStoragePoolInfoVDS method >2014-10-20 04:48:57,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [15c82c98] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to GetStoragePoolInfoVDS, error = Error storage pool action: (u'spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8',), code = 300 >2014-10-20 04:49:08,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-9) domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 in problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 04:49:10,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-17) [15c82c98] START, SpmStopVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8), log id: 17143d8b >2014-10-20 04:49:10,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-17) [15c82c98] SpmStopVDSCommand::Stopping SPM on vds monique-vds05.tlv.redhat.com, pool id 1616f9cd-9b21-400e-b38b-2354bf808db8 >2014-10-20 04:49:10,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-17) [15c82c98] FINISH, SpmStopVDSCommand, log id: 17143d8b >2014-10-20 04:49:10,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) [15c82c98] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-20 04:49:10,904 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-20 04:49:10,907 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) [1975d95f] Storage Pool 1616f9cd-9b21-400e-b38b-2354bf808db8 - Updating Storage Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 status from Active to Unknown, reason : null >2014-10-20 04:49:10,957 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [1975d95f] Correlation ID: 1975d95f, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-10-20 04:49:11,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) [1975d95f] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-20 04:49:11,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) [1975d95f] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-10-20 04:49:11,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ac8607e >2014-10-20 04:49:11,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] spmStart polling started: taskId = c7aec3c3-3432-4c7a-b752-495795f6aca6 >2014-10-20 04:49:12,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] spmStart polling ended: taskId = c7aec3c3-3432-4c7a-b752-495795f6aca6 task status = finished >2014-10-20 04:49:12,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] spmStart polling ended, spm status: SPM >2014-10-20 04:49:12,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, taskId=c7aec3c3-3432-4c7a-b752-495795f6aca6), log id: 18cc0c37 >2014-10-20 04:49:12,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] FINISH, HSMClearTaskVDSCommand, log id: 18cc0c37 >2014-10-20 04:49:12,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) [1975d95f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@464f1d73, log id: 3ac8607e >2014-10-20 04:49:12,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) [1975d95f] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-10-20 04:49:12,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [1975d95f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-10-20 04:49:12,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-33) [1975d95f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false), log id: 1f0c1683 >2014-10-20 04:49:12,190 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-20 04:49:12,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-33) [1975d95f] -- executeIrsBrokerCommand: Attempting on storage pool 1616f9cd-9b21-400e-b38b-2354bf808db8 >2014-10-20 04:49:12,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-33) [1975d95f] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 75a78b79 >2014-10-20 04:49:12,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-33) [1975d95f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75a78b79 >2014-10-20 04:49:12,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-33) [1975d95f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f0c1683 >2014-10-20 04:49:12,327 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-33) [1975d95f] Discovered no tasks on Storage Pool DC1 >2014-10-20 04:49:23,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-18) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 recovered from problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 04:49:23,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-18) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-10-20 05:43:51,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-10-20 06:43:51,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-10-20 07:43:51,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5a60ba5c] Attempting to update VMs/Templates Ovf. >2014-10-20 08:43:51,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [534fb9eb] Attempting to update VMs/Templates Ovf. >2014-10-20 09:43:51,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-10-20 10:43:51,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-10-20 11:43:51,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-10-20 12:43:51,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-10-20 13:43:51,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-10-20 14:43:51,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [701ca55e] Attempting to update VMs/Templates Ovf. >2014-10-20 15:43:51,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [10ec4770] Attempting to update VMs/Templates Ovf. >2014-10-20 16:43:51,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) [78c9ce85] Attempting to update VMs/Templates Ovf. >2014-10-20 17:43:51,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-10-20 18:43:51,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [179cd8b8] Attempting to update VMs/Templates Ovf. >2014-10-20 19:43:51,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) [1975d95f] Attempting to update VMs/Templates Ovf. >2014-10-20 20:43:51,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) [4d483c7a] Attempting to update VMs/Templates Ovf. >2014-10-20 21:43:52,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-10-20 22:43:52,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) [2d07f4c3] Attempting to update VMs/Templates Ovf. >2014-10-20 23:43:52,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [43a49e41] Attempting to update VMs/Templates Ovf. >2014-10-20 23:53:34,661 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-25) domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 in problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 23:53:49,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 recovered from problem. vds: monique-vds05.tlv.redhat.com >2014-10-20 23:53:49,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-10-20 23:54:20,250 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [3e41b5f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain SD1 experienced a high latency of 6.80115 seconds from host monique-vds01.tlv.redhat.com. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-10-20 23:54:27,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) Failed in SpmStatusVDS method >2014-10-20 23:54:27,989 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = (-202, 'Sanlock resource read failure', 'Sanlock exception'), code = 100 >2014-10-20 23:54:28,001 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-63) [28697350] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-20 23:54:28,003 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) [28697350] Storage Pool 1616f9cd-9b21-400e-b38b-2354bf808db8 - Updating Storage Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 status from Active to Unknown, reason : null >2014-10-20 23:54:28,011 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [28697350] Correlation ID: 28697350, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: General Exception). >2014-10-20 23:54:40,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) [28697350] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-20 23:54:40,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) [28697350] Found spm host d8177efa-36f1-40fa-b2d6-01cbe63b3982, host name: monique-vds01.tlv.redhat.com, according to spmId: 2. >2014-10-20 23:54:40,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) [28697350] SpmStatus on vds d8177efa-36f1-40fa-b2d6-01cbe63b3982: SPM >2014-10-20 23:54:40,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) [28697350] Using old spm server: monique-vds01.tlv.redhat.com, no start needed >2014-10-20 23:54:40,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) [28697350] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-10-20 23:54:40,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [28697350] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-10-20 23:54:40,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-24) [28697350] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false), log id: 316d07c3 >2014-10-20 23:54:40,582 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-20 23:54:40,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-24) [28697350] -- executeIrsBrokerCommand: Attempting on storage pool 1616f9cd-9b21-400e-b38b-2354bf808db8 >2014-10-20 23:54:40,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-24) [28697350] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 206a3056 >2014-10-20 23:54:40,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-24) [28697350] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 206a3056 >2014-10-20 23:54:40,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-24) [28697350] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 316d07c3 >2014-10-20 23:54:40,787 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-24) [28697350] Discovered no tasks on Storage Pool DC1 >2014-10-21 00:43:52,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-10-21 01:43:52,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-10-21 02:43:52,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) [2e4300e3] Attempting to update VMs/Templates Ovf. >2014-10-21 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Start deleteAgedOutCommandEntities >2014-10-21 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-66) Start deleteAgedOutAuditLogs >2014-10-21 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-66) Finished deleteAgedOutAuditLogs >2014-10-21 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-25) Finished deleteAgedOutCommandEntities >2014-10-21 03:43:52,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-10-21 04:43:52,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-10-21 05:43:52,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5a60ba5c] Attempting to update VMs/Templates Ovf. >2014-10-21 06:43:52,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-21 07:43:52,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-10-21 08:43:52,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-10-21 09:43:52,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-10-21 10:43:52,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-10-21 11:28:58,797 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-5) Invalid locale found in configuration: >2014-10-21 11:28:58,797 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-5) Invalid locale found in configuration: >2014-10-21 11:43:52,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-10-21 12:43:52,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) [50ac9031] Attempting to update VMs/Templates Ovf. >2014-10-21 13:43:52,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [5916475a] Attempting to update VMs/Templates Ovf. >2014-10-21 14:43:52,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [a6d9186] Attempting to update VMs/Templates Ovf. >2014-10-21 15:43:52,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-10-21 16:43:52,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-10-21 17:43:52,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5a60ba5c] Attempting to update VMs/Templates Ovf. >2014-10-21 18:43:52,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-10-21 19:43:52,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-10-21 20:43:52,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-10-21 21:43:52,430 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-10-21 22:43:52,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-10-21 23:43:52,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-10-22 00:43:52,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-10-22 01:43:52,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-10-22 02:43:52,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-10-22 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) [30c3d974] Start deleteAgedOutCommandEntities >2014-10-22 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-77) [2e4300e3] Start deleteAgedOutAuditLogs >2014-10-22 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-77) [2e4300e3] Finished deleteAgedOutAuditLogs >2014-10-22 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) [30c3d974] Finished deleteAgedOutCommandEntities >2014-10-22 03:43:52,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-10-22 04:43:52,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [179cd8b8] Attempting to update VMs/Templates Ovf. >2014-10-22 05:43:52,574 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-10-22 06:43:52,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-10-22 07:43:52,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-10-22 08:43:52,621 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) [2e4300e3] Attempting to update VMs/Templates Ovf. >2014-10-22 09:43:52,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-10-22 10:43:52,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-10-22 11:43:52,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-10-22 12:43:52,687 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-10-22 13:34:30,555 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-10-22 13:34:30,555 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-10-22 13:43:52,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) [3582a930] Attempting to update VMs/Templates Ovf. >2014-10-22 14:43:52,724 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-10-22 15:43:52,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-10-22 16:43:52,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) [3a2f7fb2] Attempting to update VMs/Templates Ovf. >2014-10-22 17:43:52,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-10-22 17:59:56,319 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-10-22 17:59:56,469 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-10-22 17:59:56,469 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-10-22 17:59:56,470 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-10-22 17:59:56,471 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-10-22 17:59:56,471 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-10-22 17:59:56,471 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-10-22 17:59:56,471 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-10-22 17:59:56,472 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-10-22 17:59:56,472 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_AJP_PORT" is "8702". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-10-22 17:59:56,473 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_HOST" is "localhost". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_PORT" is "5432". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_SECURED" is "False". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-10-22 17:59:56,474 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_USER" is "engine". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_GROUP" is "ovirt". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-10-22 17:59:56,475 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTP_PORT" is "None". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-10-22 17:59:56,476 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PERM_MAX" is "256m". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PERM_MIN" is "256m". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-10-22 17:59:56,477 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-10-22 17:59:56,478 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-10-22 17:59:56,479 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-10-22 17:59:56,479 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_NOT_INSTALLED_URL" is "/ovirt-engine/ReportsNotInstalled.html". >2014-10-22 17:59:56,479 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-10-22 17:59:56,479 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-10-22 17:59:56,479 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_RIGHTCLICK_URL" is "". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_STOP_TIME" is "10". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-10-22 17:59:56,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_USER" is "ovirt". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-10-22 17:59:56,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 46) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-10-22 18:00:16,204 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 56) Start initializing Backend >2014-10-22 18:00:17,283 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 56) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-10-22 18:00:17,859 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 56) Running ovirt-engine 3.5.0-0.17.beta.el6ev >2014-10-22 18:00:17,860 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 56) Start initializing dictionaries >2014-10-22 18:00:17,864 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 56) Finished initializing dictionaries >2014-10-22 18:00:17,865 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Start initializing AuditLogCleanupManager >2014-10-22 18:00:17,865 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-10-22 18:00:17,926 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Finished initializing AuditLogCleanupManager >2014-10-22 18:00:17,926 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Start initializing CommandEntityCleanupManager >2014-10-22 18:00:17,927 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-10-22 18:00:17,951 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 56) Finished initializing CommandEntityCleanupManager >2014-10-22 18:00:17,964 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 56) Start initializing TagsDirector >2014-10-22 18:00:17,980 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 56) Tag root added to tree >2014-10-22 18:00:18,024 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 56) Finished initializing TagsDirector >2014-10-22 18:00:18,025 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 56) Start initializing IsoDomainListSyncronizer >2014-10-22 18:00:18,050 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 56) Finished initializing IsoDomainListSyncronizer >2014-10-22 18:00:18,149 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 56) Completed initializing handlers >2014-10-22 18:00:18,235 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 56) Start initializing ErrorTranslatorImpl >2014-10-22 18:00:18,298 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 56) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-10-22 18:00:18,299 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 56) Finished initializing ErrorTranslatorImpl >2014-10-22 18:00:18,300 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 56) Start initializing ErrorTranslatorImpl >2014-10-22 18:00:18,300 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 56) Finished initializing ErrorTranslatorImpl >2014-10-22 18:00:18,300 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 56) Mark incomplete jobs as UNKNOWN >2014-10-22 18:00:18,391 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 56) Start initializing JobRepositoryCleanupManager >2014-10-22 18:00:18,392 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 56) Finished initializing JobRepositoryCleanupManager >2014-10-22 18:00:18,392 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 56) Start initializing AutoRecoveryManager >2014-10-22 18:00:18,394 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 56) Finished initializing AutoRecoveryManager >2014-10-22 18:00:18,397 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 56) Start initializing ExecutionMessageDirector >2014-10-22 18:00:18,399 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 56) Finished initializing ExecutionMessageDirector >2014-10-22 18:00:18,575 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Loading extension 'builtin-authn-internal' >2014-10-22 18:00:18,590 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Extension 'builtin-authn-internal' loaded >2014-10-22 18:00:18,605 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Loading extension 'internal' >2014-10-22 18:00:18,608 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Extension 'internal' loaded >2014-10-22 18:00:18,609 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Initializing extension 'builtin-authn-internal' >2014-10-22 18:00:18,609 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Extension 'builtin-authn-internal' initialized >2014-10-22 18:00:18,611 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Initializing extension 'internal' >2014-10-22 18:00:18,611 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Extension 'internal' initialized >2014-10-22 18:00:18,611 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Start of enabled extensions list >2014-10-22 18:00:18,612 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-22 18:00:18,612 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-22 18:00:18,612 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 56) End of enabled extensions list >2014-10-22 18:00:19,019 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 56) Start initializing DbUserCacheManager >2014-10-22 18:00:19,020 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 56) Finished initializing DbUserCacheManager >2014-10-22 18:00:19,078 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 56) Initialization of AsyncTaskManager completed successfully. >2014-10-22 18:00:19,093 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 56) Start initializing ResourceManager >2014-10-22 18:00:19,440 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Entered VdsManager constructor >2014-10-22 18:00:19,534 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-10-22 18:00:20,202 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 56) VDS 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c was added to the Resource Manager >2014-10-22 18:00:20,203 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Entered VdsManager constructor >2014-10-22 18:00:20,258 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-10-22 18:00:20,290 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 56) VDS d8177efa-36f1-40fa-b2d6-01cbe63b3982 was added to the Resource Manager >2014-10-22 18:00:20,290 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Entered VdsManager constructor >2014-10-22 18:00:20,337 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 56) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-22 18:00:20,368 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 56) VDS e9342ea1-ce56-435b-8284-0e4febfec3c1 was added to the Resource Manager >2014-10-22 18:00:20,436 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 56) Finished initializing ResourceManager >2014-10-22 18:00:20,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 56) Initialization of OvfDataUpdater completed successfully. >2014-10-22 18:00:20,463 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Start scheduling to enable vds load balancer >2014-10-22 18:00:20,463 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Finished scheduling to enable vds load balancer >2014-10-22 18:00:20,463 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Start HA Reservation check >2014-10-22 18:00:20,464 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Finished HA Reservation check >2014-10-22 18:00:20,498 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-10-22 18:00:20,521 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 56) Init VM custom properties utilities >2014-10-22 18:00:20,522 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 56) Init device custom properties utilities >2014-10-22 18:00:20,528 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Initializing Scheduling manager >2014-10-22 18:00:20,565 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-10-22 18:00:20,689 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) External scheduler disabled, discovery skipped >2014-10-22 18:00:20,689 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 56) Initialized Scheduling manager >2014-10-22 18:00:20,689 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 56) Initializing DWH Heart Beat >2014-10-22 18:00:20,690 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 56) DWH Heart Beat initialized >2014-10-22 18:00:23,981 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-22 18:00:24,387 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-22 18:00:24,454 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-22 18:00:26,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 6ad1ef20 >2014-10-22 18:00:26,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 6ad1ef20 >2014-10-22 18:00:26,914 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-22 18:00:27,285 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-22 18:00:30,782 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-22 18:00:30,808 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Storage Pool 1616f9cd-9b21-400e-b38b-2354bf808db8 - Updating Storage Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 status from Active to Unknown, reason : null >2014-10-22 18:00:30,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: UNASSIGNED not exist in string table >2014-10-22 18:00:30,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-10-22 18:00:30,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-10-22 18:00:30,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-10-22 18:00:30,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-10-22 18:00:30,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-10-22 18:00:30,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-10-22 18:00:31,614 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Correlation ID: 1e3050ae, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-10-22 18:00:32,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-22 18:00:32,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Found spm host d8177efa-36f1-40fa-b2d6-01cbe63b3982, host name: monique-vds01.tlv.redhat.com, according to spmId: 2. >2014-10-22 18:00:32,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] SpmStatus on vds d8177efa-36f1-40fa-b2d6-01cbe63b3982: SPM >2014-10-22 18:00:32,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Using old spm server: monique-vds01.tlv.redhat.com, no start needed >2014-10-22 18:00:32,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-10-22 18:00:32,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-10-22 18:00:32,805 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-22 18:00:32,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false), log id: 4a4df445 >2014-10-22 18:00:33,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] -- executeIrsBrokerCommand: Attempting on storage pool 1616f9cd-9b21-400e-b38b-2354bf808db8 >2014-10-22 18:00:33,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 346662e2 >2014-10-22 18:00:33,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 346662e2 >2014-10-22 18:00:33,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a4df445 >2014-10-22 18:00:33,156 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-6) [1e3050ae] Discovered no tasks on Storage Pool DC1 >2014-10-22 18:00:50,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 7c902656 >2014-10-22 18:00:50,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 7c902656 >2014-10-22 18:00:50,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]), log id: 44256add >2014-10-22 18:00:50,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 44256add >2014-10-22 18:00:50,832 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: monique-vds01.tlv.redhat.com >2014-10-22 18:00:50,834 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-10-22 18:00:50,852 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [71f0732f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-22 18:00:50,913 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-22 19:00:20,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-10-22 20:00:20,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-10-22 21:00:20,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-10-22 22:00:20,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-10-22 23:00:20,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-10-23 00:00:20,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Attempting to update VMs/Templates Ovf. >2014-10-23 01:00:20,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-10-23 02:00:20,688 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-10-23 03:00:20,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-10-23 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Start deleteAgedOutAuditLogs >2014-10-23 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-95) Start deleteAgedOutCommandEntities >2014-10-23 03:35:35,022 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-95) Finished deleteAgedOutCommandEntities >2014-10-23 03:35:35,026 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Finished deleteAgedOutAuditLogs >2014-10-23 04:00:20,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-10-23 05:00:20,752 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-10-23 06:00:20,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-10-23 07:00:20,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-10-23 08:00:20,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Attempting to update VMs/Templates Ovf. >2014-10-23 09:00:20,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-10-23 10:00:20,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-10-23 10:41:12,474 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 10:41:12,475 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 10:41:38,822 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-5) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-10-23 10:41:38,823 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-5) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-10-23 10:41:38,879 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-5) Invalid locale found in configuration: >2014-10-23 10:41:38,880 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-5) Invalid locale found in configuration: >2014-10-23 10:42:12,466 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginAdminUserCommand internal: false. >2014-10-23 10:42:12,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 10:42:22,596 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 10:42:22,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 10:44:06,995 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:95) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.write(SSLClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.processOutgoing(ReactorClient.java:199) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:170) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 10:44:23,567 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:95) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.write(SSLClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.processOutgoing(ReactorClient.java:199) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:170) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 10:45:01,122 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:95) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.write(SSLClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.processOutgoing(ReactorClient.java:199) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:170) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 10:47:03,806 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-82) Command ListVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:03,807 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:03,816 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:03,817 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:1000) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:949) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:667) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 10:47:03,874 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Failed to refresh VDS , vds = e9342ea1-ce56-435b-8284-0e4febfec3c1 : monique-vds05.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:07,033 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 10:47:07,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 37b7178e >2014-10-23 10:47:07,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, GetHardwareInfoVDSCommand, log id: 37b7178e >2014-10-23 10:47:07,626 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [506517d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 10:47:07,652 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [354fd24c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:47:07,716 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [661fb146] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:47:07,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [661fb146] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 477ee225 >2014-10-23 10:47:07,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [661fb146] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 477ee225 >2014-10-23 10:47:07,869 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [661fb146] Host monique-vds05.tlv.redhat.com storage connection was succeeded >2014-10-23 10:47:07,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-35) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: a809a6c >2014-10-23 10:47:07,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: a809a6c >2014-10-23 10:47:07,939 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-35) Could not connect host monique-vds05.tlv.redhat.com to pool DC1 with the message: Error storage pool connection: (u"spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, msdUUID=f548e584-2ca0-41da-9eb4-e5f6de3a4fc7, masterVersion=1, hostID=3, domainsMap={u'f548e584-2ca0-41da-9eb4-e5f6de3a4fc7': u'active'}",) >2014-10-23 10:47:08,001 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-48) [5619272c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 10:47:08,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [5619272c] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 15ed7bd2 >2014-10-23 10:47:08,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [5619272c] FINISH, SetVdsStatusVDSCommand, log id: 15ed7bd2 >2014-10-23 10:47:08,131 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5619272c] Correlation ID: 5619272c, Job ID: e10f2b2b-5ef8-49cd-ad70-802c5b439dc8, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-10-23 10:47:08,180 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5619272c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 10:47:08,208 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5619272c] Correlation ID: 354fd24c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host monique-vds05.tlv.redhat.com to Storage Pool DC1 >2014-10-23 10:47:08,240 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 10:47:08,243 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [576e6757] Host e9342ea1-ce56-435b-8284-0e4febfec3c1 : monique-vds05.tlv.redhat.com is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-10-23 10:47:19,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-37) Command ListVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:19,811 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:19,854 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:19,854 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:1000) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:949) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:667) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 10:47:19,861 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) Failed to refresh VDS , vds = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:23,425 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 10:47:23,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 4b0144dc >2014-10-23 10:47:23,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GetHardwareInfoVDSCommand, log id: 4b0144dc >2014-10-23 10:47:24,110 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [1a633010] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 10:47:24,150 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-70) [3fbe41c0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:47:24,179 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [54635ceb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:47:24,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [54635ceb] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43d1bcb8 >2014-10-23 10:47:24,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [54635ceb] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 43d1bcb8 >2014-10-23 10:47:24,265 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [54635ceb] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 10:47:24,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 17213e40 >2014-10-23 10:47:24,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 17213e40 >2014-10-23 10:47:24,375 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-21) Could not connect host rose07.qa.lab.tlv.redhat.com to pool DC1 with the message: Error storage pool connection: (u"spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, msdUUID=f548e584-2ca0-41da-9eb4-e5f6de3a4fc7, masterVersion=1, hostID=1, domainsMap={u'f548e584-2ca0-41da-9eb4-e5f6de3a4fc7': u'active'}",) >2014-10-23 10:47:24,464 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-70) [78f40174] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 10:47:24,552 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [78f40174] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 402ab346 >2014-10-23 10:47:24,583 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [78f40174] FINISH, SetVdsStatusVDSCommand, log id: 402ab346 >2014-10-23 10:47:24,622 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [78f40174] Correlation ID: 78f40174, Job ID: ccf50547-0a70-4fec-8e2f-7ba7c1b536f9, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-10-23 10:47:24,769 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [78f40174] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.There is no other host in the data center that can be used to test the power management settings. >2014-10-23 10:47:24,797 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [78f40174] Correlation ID: 3fbe41c0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rose07.qa.lab.tlv.redhat.com to Storage Pool DC1 >2014-10-23 10:47:24,859 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [21138393] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 10:47:24,861 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [21138393] Host 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-10-23 10:47:57,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:57,874 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:57,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) Command SpmStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:57,903 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-62) [1e809b26] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:47:57,908 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-10-23 10:47:57,908 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:1000) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:949) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:667) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 10:47:57,910 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-62) [1e809b26] Storage Pool 1616f9cd-9b21-400e-b38b-2354bf808db8 - Updating Storage Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 status from Active to Unknown, reason : null >2014-10-23 10:47:57,960 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = d8177efa-36f1-40fa-b2d6-01cbe63b3982 : monique-vds01.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 10:47:57,968 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [1e809b26] Correlation ID: 1e809b26, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host monique-vds01.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-10-23 10:47:58,104 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 10:47:58,240 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [1e809b26] Failed in GetStoragePoolInfoVDS method >2014-10-23 10:47:58,241 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) [1e809b26] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to GetStoragePoolInfoVDS, error = Unknown pool id, pool not connected: (u'1616f9cd-9b21-400e-b38b-2354bf808db8',), code = 309 >2014-10-23 10:48:00,967 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 10:48:01,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]), log id: 7dd8132f >2014-10-23 10:48:01,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GetHardwareInfoVDSCommand, log id: 7dd8132f >2014-10-23 10:48:01,785 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [fd78e4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 10:48:01,797 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-35) [525f0944] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:48:01,815 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:48:01,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40828641 >2014-10-23 10:48:01,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 40828641 >2014-10-23 10:48:01,858 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] Host monique-vds01.tlv.redhat.com storage connection was succeeded >2014-10-23 10:48:01,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-3) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1643f234 >2014-10-23 10:48:01,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 1643f234 >2014-10-23 10:48:01,942 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-3) Could not connect host monique-vds01.tlv.redhat.com to pool DC1, as the master domain is in inactive/unknown status - not failing the operation >2014-10-23 10:48:08,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:48:08,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:48:08,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:48:27,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 47a9b65c >2014-10-23 10:48:27,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-35) [4c671d9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 47a9b65c >2014-10-23 10:48:27,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [4c671d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-10-23 10:48:27,164 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [4c671d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-10-23 10:48:27,233 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [7090c5eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 10:48:27,348 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-47) domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 in problem. vds: monique-vds01.tlv.redhat.com >2014-10-23 10:48:28,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:48:28,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3740da0 >2014-10-23 10:48:28,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, ConnectStoragePoolVDSCommand, log id: 3740da0 >2014-10-23 10:48:28,468 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:48:38,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:48:38,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:48:38,519 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:48:38,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:48:38,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3684d9b0 >2014-10-23 10:48:38,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, ConnectStoragePoolVDSCommand, log id: 3684d9b0 >2014-10-23 10:48:38,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:48:48,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:48:48,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:48:48,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:48:48,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:48:48,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6552ab6 >2014-10-23 10:48:48,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 6552ab6 >2014-10-23 10:48:48,796 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:48:58,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:48:58,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:48:58,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:48:58,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:48:58,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3fc4441 >2014-10-23 10:48:58,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, ConnectStoragePoolVDSCommand, log id: 3fc4441 >2014-10-23 10:48:58,924 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:08,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:08,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:08,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:08,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:08,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2ed75522 >2014-10-23 10:49:09,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, ConnectStoragePoolVDSCommand, log id: 2ed75522 >2014-10-23 10:49:09,045 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:19,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:19,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:19,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:19,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:19,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 368ba5e >2014-10-23 10:49:19,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, ConnectStoragePoolVDSCommand, log id: 368ba5e >2014-10-23 10:49:19,171 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:29,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:29,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:29,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:29,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-87) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:29,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 772375d >2014-10-23 10:49:29,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, ConnectStoragePoolVDSCommand, log id: 772375d >2014-10-23 10:49:29,284 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:39,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:39,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:39,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:39,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:39,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4b2b815c >2014-10-23 10:49:39,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, ConnectStoragePoolVDSCommand, log id: 4b2b815c >2014-10-23 10:49:39,395 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:49,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:49,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:49,433 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:49,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:49,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 76569e3 >2014-10-23 10:49:49,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, ConnectStoragePoolVDSCommand, log id: 76569e3 >2014-10-23 10:49:49,541 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:49:59,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:49:59,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:49:59,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:49:59,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-72) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:49:59,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4488a218 >2014-10-23 10:49:59,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, ConnectStoragePoolVDSCommand, log id: 4488a218 >2014-10-23 10:49:59,702 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-72) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:00,061 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-2) Autorecovering 2 hosts >2014-10-23 10:50:00,062 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-2) Autorecovering hosts id: e9342ea1-ce56-435b-8284-0e4febfec3c1, name : monique-vds05.tlv.redhat.com >2014-10-23 10:50:00,067 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Lock Acquired to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 10:50:00,068 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 10:50:00,068 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Before acquiring lock in order to prevent monitoring for host monique-vds05.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,068 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Lock acquired, from now a monitoring of host will be skipped for host monique-vds05.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,092 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3fa59f44 >2014-10-23 10:50:00,134 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] FINISH, SetVdsStatusVDSCommand, log id: 3fa59f44 >2014-10-23 10:50:00,256 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Activate finished. Lock released. Monitoring can run now for host monique-vds05.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [37c097a0] Correlation ID: 37c097a0, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was autorecovered. >2014-10-23 10:50:00,268 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [37c097a0] Lock freed to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 10:50:00,269 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-2) [37c097a0] Autorecovering hosts id: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, name : rose07.qa.lab.tlv.redhat.com >2014-10-23 10:50:00,321 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Lock Acquired to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 10:50:00,321 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 10:50:00,321 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,321 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,356 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1c73dc64 >2014-10-23 10:50:00,361 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, SetVdsStatusVDSCommand, log id: 1c73dc64 >2014-10-23 10:50:00,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 41acdae7 >2014-10-23 10:50:00,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 41acdae7 >2014-10-23 10:50:00,440 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 10:50:00,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [38848d6b] Correlation ID: 38848d6b, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was autorecovered. >2014-10-23 10:50:00,448 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Lock freed to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 10:50:02,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 620f28ce >2014-10-23 10:50:02,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GetHardwareInfoVDSCommand, log id: 620f28ce >2014-10-23 10:50:02,642 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [176021cd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 10:50:02,657 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-96) [43a1ed87] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:50:02,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:50:02,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 642ba185 >2014-10-23 10:50:02,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 642ba185 >2014-10-23 10:50:02,696 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] Host monique-vds05.tlv.redhat.com storage connection was succeeded >2014-10-23 10:50:02,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-34) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3a99e7a6 >2014-10-23 10:50:02,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-34) FINISH, ConnectStoragePoolVDSCommand, log id: 3a99e7a6 >2014-10-23 10:50:02,753 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-34) Could not connect host monique-vds05.tlv.redhat.com to pool DC1, as the master domain is in inactive/unknown status - not failing the operation >2014-10-23 10:50:02,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1), log id: 71a95d8d >2014-10-23 10:50:02,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [477fa9fb] FINISH, SetMOMPolicyParametersVDSCommand, log id: 71a95d8d >2014-10-23 10:50:02,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [477fa9fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-10-23 10:50:02,843 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [477fa9fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 10:50:02,857 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [3bc59a19] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 10:50:03,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 41d4ade7 >2014-10-23 10:50:03,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GetHardwareInfoVDSCommand, log id: 41d4ade7 >2014-10-23 10:50:03,280 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [9fc61c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 10:50:03,289 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-87) [4bc5236d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:50:03,297 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 10:50:03,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 323bd46d >2014-10-23 10:50:03,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 323bd46d >2014-10-23 10:50:03,341 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 10:50:03,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2b2b38b4 >2014-10-23 10:50:03,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 2b2b38b4 >2014-10-23 10:50:03,407 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-2) Could not connect host rose07.qa.lab.tlv.redhat.com to pool DC1, as the master domain is in inactive/unknown status - not failing the operation >2014-10-23 10:50:09,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:50:09,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:50:09,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:50:09,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:09,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 78b1c53 >2014-10-23 10:50:09,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: 78b1c53 >2014-10-23 10:50:09,848 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:09,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:50:09,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:09,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: b58fd41 >2014-10-23 10:50:09,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: b58fd41 >2014-10-23 10:50:09,916 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:19,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:50:19,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:50:19,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:50:19,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:19,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7a557715 >2014-10-23 10:50:20,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7a557715 >2014-10-23 10:50:20,040 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:20,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:50:20,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:20,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2a1ea403 >2014-10-23 10:50:20,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2a1ea403 >2014-10-23 10:50:20,107 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:28,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [bdc6329] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 10:50:28,713 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) [bdc6329] Using Host monique-vds05.tlv.redhat.com from cluster cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 10:50:28,753 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-87) [bdc6329] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 10:50:28,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, targetVdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1e5291cf >2014-10-23 10:50:28,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1e5291cf >2014-10-23 10:50:28,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 63148a50 >2014-10-23 10:50:28,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-87) [bdc6329] FINISH, SetMOMPolicyParametersVDSCommand, log id: 63148a50 >2014-10-23 10:50:28,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [bdc6329] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 10:50:28,991 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [bdc6329] Correlation ID: 4bc5236d, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-10-23 10:50:29,071 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [23d50e56] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 10:50:30,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:50:30,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:50:30,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:50:30,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:30,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2571655e >2014-10-23 10:50:30,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 2571655e >2014-10-23 10:50:30,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:30,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:50:30,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:30,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 14780c8e >2014-10-23 10:50:30,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 14780c8e >2014-10-23 10:50:30,431 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:30,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:50:30,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:30,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5784e12f >2014-10-23 10:50:30,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 5784e12f >2014-10-23 10:50:30,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:40,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:50:40,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:50:40,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:50:40,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:40,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 63c3f540 >2014-10-23 10:50:40,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, ConnectStoragePoolVDSCommand, log id: 63c3f540 >2014-10-23 10:50:40,857 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:40,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:50:40,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:40,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 73d0d95a >2014-10-23 10:50:41,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, ConnectStoragePoolVDSCommand, log id: 73d0d95a >2014-10-23 10:50:41,029 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:41,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:50:41,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:41,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7042ac8 >2014-10-23 10:50:41,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, ConnectStoragePoolVDSCommand, log id: 7042ac8 >2014-10-23 10:50:41,184 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:51,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:50:51,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:50:51,329 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:50:51,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:51,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 10dce49f >2014-10-23 10:50:51,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 10dce49f >2014-10-23 10:50:51,461 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:51,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:50:51,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:51,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2c0b6462 >2014-10-23 10:50:51,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 2c0b6462 >2014-10-23 10:50:51,613 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:50:51,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:50:51,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:50:51,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2be53536 >2014-10-23 10:50:51,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 2be53536 >2014-10-23 10:50:51,764 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:01,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:01,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:51:01,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:01,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:01,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 47b8df9e >2014-10-23 10:51:02,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 47b8df9e >2014-10-23 10:51:02,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:02,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:51:02,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:02,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7392c88a >2014-10-23 10:51:02,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 7392c88a >2014-10-23 10:51:02,180 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:02,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:51:02,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:02,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 103a4dad >2014-10-23 10:51:02,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 103a4dad >2014-10-23 10:51:02,298 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:12,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:12,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:51:12,385 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:12,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:12,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3e5ace45 >2014-10-23 10:51:12,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, ConnectStoragePoolVDSCommand, log id: 3e5ace45 >2014-10-23 10:51:12,501 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:12,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:51:12,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:12,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 72b1d59d >2014-10-23 10:51:12,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, ConnectStoragePoolVDSCommand, log id: 72b1d59d >2014-10-23 10:51:12,649 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:12,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:51:12,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:12,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 38e4bd93 >2014-10-23 10:51:12,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, ConnectStoragePoolVDSCommand, log id: 38e4bd93 >2014-10-23 10:51:12,836 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:22,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:22,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:51:22,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:23,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:23,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4cab7b84 >2014-10-23 10:51:23,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 4cab7b84 >2014-10-23 10:51:23,067 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:23,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:51:23,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:23,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 532035d5 >2014-10-23 10:51:23,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 532035d5 >2014-10-23 10:51:23,230 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:23,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:51:23,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:23,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 792b9687 >2014-10-23 10:51:23,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 792b9687 >2014-10-23 10:51:23,399 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:33,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:33,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:51:33,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:33,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:33,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 488d0a6b >2014-10-23 10:51:33,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 488d0a6b >2014-10-23 10:51:33,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:33,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:51:33,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:33,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 79f25604 >2014-10-23 10:51:33,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 79f25604 >2014-10-23 10:51:33,811 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:33,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:51:33,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:33,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 544687ee >2014-10-23 10:51:33,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 544687ee >2014-10-23 10:51:33,963 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:44,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:44,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:51:44,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:44,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:44,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5563273b >2014-10-23 10:51:44,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 5563273b >2014-10-23 10:51:44,175 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:44,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:51:44,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:44,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6a66350b >2014-10-23 10:51:44,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 6a66350b >2014-10-23 10:51:44,441 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:44,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:51:44,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:44,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: bd36e41 >2014-10-23 10:51:44,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: bd36e41 >2014-10-23 10:51:44,597 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:54,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:51:54,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:51:54,757 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:51:54,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:54,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 15956d82 >2014-10-23 10:51:54,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, ConnectStoragePoolVDSCommand, log id: 15956d82 >2014-10-23 10:51:54,843 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:54,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:51:54,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:54,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 667a1355 >2014-10-23 10:51:55,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, ConnectStoragePoolVDSCommand, log id: 667a1355 >2014-10-23 10:51:55,015 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:51:55,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:51:55,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:51:55,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3baea72e >2014-10-23 10:51:55,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, ConnectStoragePoolVDSCommand, log id: 3baea72e >2014-10-23 10:51:55,175 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:05,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:05,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:52:05,271 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:05,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:05,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4e882995 >2014-10-23 10:52:05,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 4e882995 >2014-10-23 10:52:05,384 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:05,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:52:05,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:05,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 312292a0 >2014-10-23 10:52:05,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 312292a0 >2014-10-23 10:52:05,566 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:05,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:05,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:05,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 292c0eab >2014-10-23 10:52:05,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 292c0eab >2014-10-23 10:52:05,692 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:15,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:15,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:52:15,780 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:15,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:15,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4b328309 >2014-10-23 10:52:15,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, ConnectStoragePoolVDSCommand, log id: 4b328309 >2014-10-23 10:52:15,866 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:15,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:52:15,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:15,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6b717b6c >2014-10-23 10:52:16,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, ConnectStoragePoolVDSCommand, log id: 6b717b6c >2014-10-23 10:52:16,023 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:16,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:16,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:16,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 57b042e1 >2014-10-23 10:52:16,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, ConnectStoragePoolVDSCommand, log id: 57b042e1 >2014-10-23 10:52:16,188 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:26,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:26,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:52:26,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:26,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:26,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 77073611 >2014-10-23 10:52:26,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 77073611 >2014-10-23 10:52:26,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:26,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:52:26,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:26,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 24d6d03a >2014-10-23 10:52:26,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 24d6d03a >2014-10-23 10:52:26,591 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:26,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:26,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:26,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 78f85501 >2014-10-23 10:52:26,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 78f85501 >2014-10-23 10:52:26,785 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:34,574 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [4fd372ab] Lock Acquired to object EngineLock [exclusiveLocks= key: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 value: STORAGE >, sharedLocks= ] >2014-10-23 10:52:34,658 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:52:34,792 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Lock freed to object EngineLock [exclusiveLocks= key: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 value: STORAGE >, sharedLocks= ] >2014-10-23 10:52:34,792 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] ActivateStorage Domain. Before Connect all hosts to pool. Time:10/23/14 10:52 AM >2014-10-23 10:52:35,047 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-1) [675b37fb] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:52:35,048 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-20) [7cd5618c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:52:35,049 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-35) [1cf76e5b] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:52:35,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-20) [7cd5618c] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 141a3afe >2014-10-23 10:52:35,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-35) [1cf76e5b] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 21d89299 >2014-10-23 10:52:35,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-20) [7cd5618c] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 141a3afe >2014-10-23 10:52:35,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-35) [1cf76e5b] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 21d89299 >2014-10-23 10:52:35,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-1) [675b37fb] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2c993164 >2014-10-23 10:52:35,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-1) [675b37fb] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 2c993164 >2014-10-23 10:52:35,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] START, ActivateStorageDomainVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false, storageDomainId = f548e584-2ca0-41da-9eb4-e5f6de3a4fc7), log id: 19cebc54 >2014-10-23 10:52:35,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:35,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 708b7f21 >2014-10-23 10:52:35,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] FINISH, ConnectStoragePoolVDSCommand, log id: 708b7f21 >2014-10-23 10:52:35,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:35,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:35,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:35,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7004a842 >2014-10-23 10:52:35,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] FINISH, ConnectStoragePoolVDSCommand, log id: 7004a842 >2014-10-23 10:52:35,701 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:35,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:52:35,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:35,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 366b1924 >2014-10-23 10:52:36,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] FINISH, ConnectStoragePoolVDSCommand, log id: 366b1924 >2014-10-23 10:52:36,016 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:36,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] FINISH, ActivateStorageDomainVDSCommand, log id: 19cebc54 >2014-10-23 10:52:36,017 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001) >2014-10-23 10:52:36,094 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Command [id=360b3588-842a-4984-903b-4bbc671867cc]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageId = f548e584-2ca0-41da-9eb4-e5f6de3a4fc7, status=Unknown]. >2014-10-23 10:52:36,185 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-32) [4fd372ab] Correlation ID: 4fd372ab, Job ID: 6fdf489c-93ad-42d4-9828-b7f4b5af9501, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain SD1 (Data Center DC1) by admin >2014-10-23 10:52:36,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:36,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:52:36,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:36,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:36,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1e294eb8 >2014-10-23 10:52:36,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 1e294eb8 >2014-10-23 10:52:36,988 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:37,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:52:37,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:37,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 36199e3f >2014-10-23 10:52:37,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 36199e3f >2014-10-23 10:52:37,305 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:37,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:52:37,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:37,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 57d8e46a >2014-10-23 10:52:37,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 57d8e46a >2014-10-23 10:52:37,704 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:47,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:47,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:52:47,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:47,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:47,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: e0e5831 >2014-10-23 10:52:47,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, ConnectStoragePoolVDSCommand, log id: e0e5831 >2014-10-23 10:52:47,915 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:47,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:48,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:48,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 50b92cd1 >2014-10-23 10:52:48,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, ConnectStoragePoolVDSCommand, log id: 50b92cd1 >2014-10-23 10:52:48,051 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:48,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:52:48,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:48,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 365f3a38 >2014-10-23 10:52:48,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, ConnectStoragePoolVDSCommand, log id: 365f3a38 >2014-10-23 10:52:48,199 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:58,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:52:58,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:52:58,322 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:52:58,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:58,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: ad7f7ef >2014-10-23 10:52:58,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, ConnectStoragePoolVDSCommand, log id: ad7f7ef >2014-10-23 10:52:58,411 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:58,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:52:58,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:58,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5b557970 >2014-10-23 10:52:58,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, ConnectStoragePoolVDSCommand, log id: 5b557970 >2014-10-23 10:52:58,557 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:52:58,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:52:58,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:52:58,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7916294f >2014-10-23 10:52:58,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, ConnectStoragePoolVDSCommand, log id: 7916294f >2014-10-23 10:52:58,723 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:08,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:53:08,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:53:08,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:53:08,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:08,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3dc7e7f7 >2014-10-23 10:53:09,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 3dc7e7f7 >2014-10-23 10:53:09,006 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:09,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:53:09,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:09,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 97e5710 >2014-10-23 10:53:09,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 97e5710 >2014-10-23 10:53:09,123 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:09,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:53:09,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:09,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 16d8f79d >2014-10-23 10:53:09,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 16d8f79d >2014-10-23 10:53:09,313 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:19,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:53:19,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:53:19,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:53:19,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:19,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2e194e70 >2014-10-23 10:53:19,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 2e194e70 >2014-10-23 10:53:19,541 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:19,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:53:19,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:19,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3979b5ec >2014-10-23 10:53:19,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 3979b5ec >2014-10-23 10:53:19,736 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:19,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:53:19,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:19,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2701ab5 >2014-10-23 10:53:19,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 2701ab5 >2014-10-23 10:53:19,985 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:27,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-9) starting processDomainRecovery for domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 >2014-10-23 10:53:27,475 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-9) Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7:SD1 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-10-23 10:53:27,558 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 Type: Storage >2014-10-23 10:53:27,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] START, ResetIrsVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, ignoreStopFailed = true), log id: 4380f925 >2014-10-23 10:53:27,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] START, SpmStopVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8), log id: 42bf05fc >2014-10-23 10:53:27,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:53:27,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:53:27,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:53:27,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] FINISH, SpmStopVDSCommand, log id: 42bf05fc >2014-10-23 10:53:27,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] FINISH, ResetIrsVDSCommand, log id: 4380f925 >2014-10-23 10:53:27,623 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Correlation ID: 38e20e1f, Job ID: a3f23b4e-d721-479b-8f87-5eaf6f107125, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center DC1. >2014-10-23 10:53:27,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Removing vds [] from the domain in maintenance cache >2014-10-23 10:53:27,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-9) [38e20e1f] Removing host(s) [] from hosts unseen domain report cache >2014-10-23 10:53:27,630 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-12) Finished reconstruct for pool 1616f9cd-9b21-400e-b38b-2354bf808db8. Clearing event queue >2014-10-23 10:53:30,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:30,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3b1e5208 >2014-10-23 10:53:30,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 3b1e5208 >2014-10-23 10:53:30,256 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:30,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:53:30,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:30,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: e4146d6 >2014-10-23 10:53:30,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: e4146d6 >2014-10-23 10:53:30,432 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:30,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:53:30,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:30,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 640fffc1 >2014-10-23 10:53:30,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 640fffc1 >2014-10-23 10:53:30,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:33,777 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-10) [1ef8e377] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-10-23 10:53:33,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [1ef8e377] Correlation ID: 1ef8e377, Call Stack: null, Custom Event ID: -1, Message: Event log message 34 was removed by User admin. >2014-10-23 10:53:36,738 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-3) [158235f3] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-10-23 10:53:36,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [158235f3] Correlation ID: 158235f3, Call Stack: null, Custom Event ID: -1, Message: Event log message 228 was removed by User admin. >2014-10-23 10:53:40,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:53:40,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:53:40,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:53:40,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:40,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6e729456 >2014-10-23 10:53:40,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, ConnectStoragePoolVDSCommand, log id: 6e729456 >2014-10-23 10:53:40,882 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:40,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:53:41,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:41,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3e687ac5 >2014-10-23 10:53:41,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, ConnectStoragePoolVDSCommand, log id: 3e687ac5 >2014-10-23 10:53:41,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:41,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:53:41,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:41,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 423c765b >2014-10-23 10:53:41,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, ConnectStoragePoolVDSCommand, log id: 423c765b >2014-10-23 10:53:41,331 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:41,466 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-5) [7c7e3146] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-10-23 10:53:41,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [7c7e3146] Correlation ID: 7c7e3146, Call Stack: null, Custom Event ID: -1, Message: Event log message 594 was removed by User admin. >2014-10-23 10:53:43,239 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-8) [1e57c46e] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-10-23 10:53:43,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [1e57c46e] Correlation ID: 1e57c46e, Call Stack: null, Custom Event ID: -1, Message: Event log message 605 was removed by User admin. >2014-10-23 10:53:51,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:53:51,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:53:51,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:53:51,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:51,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5310a5c1 >2014-10-23 10:53:51,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, ConnectStoragePoolVDSCommand, log id: 5310a5c1 >2014-10-23 10:53:51,605 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:51,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:53:51,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:51,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 37b46f35 >2014-10-23 10:53:51,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, ConnectStoragePoolVDSCommand, log id: 37b46f35 >2014-10-23 10:53:51,729 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:51,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:53:51,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:53:51,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 52b460c1 >2014-10-23 10:53:51,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, ConnectStoragePoolVDSCommand, log id: 52b460c1 >2014-10-23 10:53:51,910 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:53:52,019 INFO [org.ovirt.engine.core.bll.RemoveAuditLogByIdCommand] (ajp-/127.0.0.1:8702-6) [647b5a60] Running command: RemoveAuditLogByIdCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group AUDIT_LOG_MANAGEMENT with role type ADMIN >2014-10-23 10:53:52,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [647b5a60] Correlation ID: 647b5a60, Call Stack: null, Custom Event ID: -1, Message: Event log message 690 was removed by User admin. >2014-10-23 10:54:02,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:02,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:54:02,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:02,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:02,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6647b070 >2014-10-23 10:54:02,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 6647b070 >2014-10-23 10:54:02,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:02,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:54:02,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:02,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7f7729b8 >2014-10-23 10:54:02,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 7f7729b8 >2014-10-23 10:54:02,404 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:02,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:54:02,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:02,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 18d3f22a >2014-10-23 10:54:02,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 18d3f22a >2014-10-23 10:54:02,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:12,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:12,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:54:12,647 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:12,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:12,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 46cf1797 >2014-10-23 10:54:12,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 46cf1797 >2014-10-23 10:54:12,738 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:12,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:54:12,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:12,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7ce48e9d >2014-10-23 10:54:12,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 7ce48e9d >2014-10-23 10:54:12,934 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:13,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:54:13,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:13,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2bb53b7b >2014-10-23 10:54:13,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 2bb53b7b >2014-10-23 10:54:13,161 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:23,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:23,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:54:23,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:23,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:23,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 75996bc7 >2014-10-23 10:54:23,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 75996bc7 >2014-10-23 10:54:23,406 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:23,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:54:23,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:23,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 53bd0122 >2014-10-23 10:54:23,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 53bd0122 >2014-10-23 10:54:23,574 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:23,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:54:23,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:23,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 70ab4053 >2014-10-23 10:54:23,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 70ab4053 >2014-10-23 10:54:23,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:28,708 ERROR [org.ovirt.engine.core.bll.GetImagesListQuery] (ajp-/127.0.0.1:8702-6) Query GetImagesListQuery failed. Exception message is VdcBLLException: GetIsoListError (Failed with error GetIsoListError and code 314) : org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: GetIsoListError (Failed with error GetIsoListError and code 314): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: GetIsoListError (Failed with error GetIsoListError and code 314) > at org.ovirt.engine.core.bll.IsoDomainListSyncronizer.getUserRequestForStorageDomainRepoFileList(IsoDomainListSyncronizer.java:188) [bll.jar:] > at org.ovirt.engine.core.bll.GetImagesListQueryBase.getUserRequestForStorageDomainRepoFileList(GetImagesListQueryBase.java:28) [bll.jar:] > at org.ovirt.engine.core.bll.GetImagesListQueryBase.executeQueryCommand(GetImagesListQueryBase.java:19) [bll.jar:] > at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:73) [bll.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:492) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:466) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:73) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view8.runQuery(Unknown Source) [common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runQuery(GenericApiGWTServiceImpl.java:56) > at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.aaa.filters.SessionMgmtFilter.doFilter(SessionMgmtFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:512) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] > >2014-10-23 10:54:33,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:33,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:54:33,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-93) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:33,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:33,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 78c0879f >2014-10-23 10:54:34,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 78c0879f >2014-10-23 10:54:34,008 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:34,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:54:34,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:34,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6e68e64 >2014-10-23 10:54:34,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 6e68e64 >2014-10-23 10:54:34,238 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:34,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:54:34,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:34,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 50f29e3c >2014-10-23 10:54:34,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, ConnectStoragePoolVDSCommand, log id: 50f29e3c >2014-10-23 10:54:34,439 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:44,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:44,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:54:44,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:44,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:44,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 53388d13 >2014-10-23 10:54:44,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 53388d13 >2014-10-23 10:54:44,664 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:44,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:54:44,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:44,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 688b65f >2014-10-23 10:54:44,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 688b65f >2014-10-23 10:54:44,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:44,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:54:44,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:44,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 23ad7f22 >2014-10-23 10:54:44,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 23ad7f22 >2014-10-23 10:54:44,980 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:49,805 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-5) [17f29510] Lock Acquired to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 10:54:49,805 WARN [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-5) [17f29510] CanDoAction of action RemoveDisk failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_OVF_DISK_NOT_IN_APPLICABLE_STATUS,$status ILLEGAL >2014-10-23 10:54:49,805 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-5) [17f29510] Lock freed to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 10:54:55,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:54:55,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-1) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:54:55,075 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:54:55,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:55,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7fbef017 >2014-10-23 10:54:55,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, ConnectStoragePoolVDSCommand, log id: 7fbef017 >2014-10-23 10:54:55,162 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-1) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:55,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:54:55,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:55,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6bb82a92 >2014-10-23 10:54:55,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, ConnectStoragePoolVDSCommand, log id: 6bb82a92 >2014-10-23 10:54:55,317 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-1) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:54:55,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:54:55,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:54:55,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 599ff9f4 >2014-10-23 10:54:55,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, ConnectStoragePoolVDSCommand, log id: 599ff9f4 >2014-10-23 10:54:55,567 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-1) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [3bc59a19] Autorecovering 1 storage domains >2014-10-23 10:55:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [3bc59a19] Autorecovering storage domains id: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 >2014-10-23 10:55:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-96) [6c1fc222] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 Type: Storage >2014-10-23 10:55:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-96) [6c1fc222] ConnectDomainToStorage. Before Connect all hosts to pool. Time:10/23/14 10:55 AM >2014-10-23 10:55:00,056 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-29) [7ad1119d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:55:00,070 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-40) [6715740b] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:55:00,074 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-11) [6b887e98] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 10:55:00,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-11) [6b887e98] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 682a65e >2014-10-23 10:55:00,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-40) [6715740b] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20ae1c2c >2014-10-23 10:55:00,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-11) [6b887e98] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 682a65e >2014-10-23 10:55:00,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-40) [6715740b] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 20ae1c2c >2014-10-23 10:55:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [7ad1119d] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1c477250 >2014-10-23 10:55:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [7ad1119d] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 1c477250 >2014-10-23 10:55:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-96) [6c1fc222] ConnectDomainToStorage. After Connect all hosts to pool. Time:10/23/14 10:55 AM >2014-10-23 10:55:02,485 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [1980409c] Lock Acquired to object EngineLock [exclusiveLocks= key: b5d2305b-721b-491e-9ed7-ffc51c494e19 value: DISK >, sharedLocks= ] >2014-10-23 10:55:02,486 WARN [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [1980409c] CanDoAction of action RemoveDisk failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_OVF_DISK_NOT_IN_APPLICABLE_STATUS,$status ILLEGAL >2014-10-23 10:55:02,487 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [1980409c] Lock freed to object EngineLock [exclusiveLocks= key: b5d2305b-721b-491e-9ed7-ffc51c494e19 value: DISK >, sharedLocks= ] >2014-10-23 10:55:05,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:05,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:55:05,725 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:05,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:05,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6565eb6b >2014-10-23 10:55:05,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 6565eb6b >2014-10-23 10:55:05,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:05,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:55:05,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:05,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 53a570ec >2014-10-23 10:55:06,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 53a570ec >2014-10-23 10:55:06,019 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:06,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:55:06,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:06,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 52cc62a2 >2014-10-23 10:55:06,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 52cc62a2 >2014-10-23 10:55:06,239 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:16,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:16,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:55:16,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:16,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:16,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 44f3eec2 >2014-10-23 10:55:16,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 44f3eec2 >2014-10-23 10:55:16,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:16,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:55:16,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:16,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1bbf0063 >2014-10-23 10:55:16,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 1bbf0063 >2014-10-23 10:55:16,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:16,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:55:16,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:16,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6809a73d >2014-10-23 10:55:16,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 6809a73d >2014-10-23 10:55:16,750 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:26,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:26,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:55:26,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:26,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:26,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5d61b771 >2014-10-23 10:55:27,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, ConnectStoragePoolVDSCommand, log id: 5d61b771 >2014-10-23 10:55:27,046 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:27,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:55:27,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:27,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 52d319cd >2014-10-23 10:55:27,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, ConnectStoragePoolVDSCommand, log id: 52d319cd >2014-10-23 10:55:27,208 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:27,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:55:27,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:27,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 787ffc8c >2014-10-23 10:55:27,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, ConnectStoragePoolVDSCommand, log id: 787ffc8c >2014-10-23 10:55:27,445 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:37,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:37,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:55:37,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:37,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:37,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3ea82c56 >2014-10-23 10:55:37,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 3ea82c56 >2014-10-23 10:55:37,738 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:37,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:55:37,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:37,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 65b7ec7b >2014-10-23 10:55:37,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 65b7ec7b >2014-10-23 10:55:37,910 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:37,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:55:38,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:38,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5d9c9e92 >2014-10-23 10:55:38,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 5d9c9e92 >2014-10-23 10:55:38,039 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:48,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:48,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:55:48,126 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:48,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:48,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3630ba17 >2014-10-23 10:55:48,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 3630ba17 >2014-10-23 10:55:48,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:48,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:55:48,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:48,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5991d072 >2014-10-23 10:55:48,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 5991d072 >2014-10-23 10:55:48,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:48,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:55:48,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:48,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 480db4a3 >2014-10-23 10:55:48,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 480db4a3 >2014-10-23 10:55:48,519 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:58,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:55:58,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:55:58,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:55:58,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:58,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2f511f70 >2014-10-23 10:55:58,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: 2f511f70 >2014-10-23 10:55:58,705 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:58,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:55:58,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:58,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 272deb04 >2014-10-23 10:55:58,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: 272deb04 >2014-10-23 10:55:58,851 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:55:58,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:55:58,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:55:58,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: d4ca527 >2014-10-23 10:55:58,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: d4ca527 >2014-10-23 10:55:59,000 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:09,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:56:09,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:56:09,126 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:56:09,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:09,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5c0eaa5e >2014-10-23 10:56:09,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 5c0eaa5e >2014-10-23 10:56:09,239 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:09,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:56:09,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:09,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4341a6f5 >2014-10-23 10:56:09,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 4341a6f5 >2014-10-23 10:56:09,405 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:09,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:56:09,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [38848d6b] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:09,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3e7b385c >2014-10-23 10:56:09,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] FINISH, ConnectStoragePoolVDSCommand, log id: 3e7b385c >2014-10-23 10:56:09,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [38848d6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:19,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:56:19,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:56:19,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:56:19,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:19,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7358eec1 >2014-10-23 10:56:19,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 7358eec1 >2014-10-23 10:56:19,709 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:19,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:56:19,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:19,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: eca40b1 >2014-10-23 10:56:19,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: eca40b1 >2014-10-23 10:56:19,867 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:19,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:56:19,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [1e3050ae] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:19,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6e3c96 >2014-10-23 10:56:20,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] FINISH, ConnectStoragePoolVDSCommand, log id: 6e3c96 >2014-10-23 10:56:20,015 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [1e3050ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:30,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:56:30,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:56:30,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:56:30,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:30,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 27742cfa >2014-10-23 10:56:30,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 27742cfa >2014-10-23 10:56:30,235 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:30,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:56:30,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:30,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7968843b >2014-10-23 10:56:30,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 7968843b >2014-10-23 10:56:30,388 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:30,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:56:30,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:30,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6255108b >2014-10-23 10:56:30,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 6255108b >2014-10-23 10:56:30,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:40,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:56:40,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:56:40,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:56:40,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:40,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4b99890 >2014-10-23 10:56:40,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, ConnectStoragePoolVDSCommand, log id: 4b99890 >2014-10-23 10:56:40,766 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-65) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:40,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:56:40,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:40,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5ae0ceca >2014-10-23 10:56:40,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, ConnectStoragePoolVDSCommand, log id: 5ae0ceca >2014-10-23 10:56:40,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-65) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:41,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:56:41,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:41,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6d3ae35b >2014-10-23 10:56:41,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, ConnectStoragePoolVDSCommand, log id: 6d3ae35b >2014-10-23 10:56:41,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-65) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:51,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:56:51,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:56:51,218 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:56:51,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:51,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5c9bece0 >2014-10-23 10:56:51,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 5c9bece0 >2014-10-23 10:56:51,361 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:51,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:56:51,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:51,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 23d3f86a >2014-10-23 10:56:51,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 23d3f86a >2014-10-23 10:56:51,515 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:56:51,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:56:51,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [576e6757] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:56:51,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7c4b57ac >2014-10-23 10:56:51,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, ConnectStoragePoolVDSCommand, log id: 7c4b57ac >2014-10-23 10:56:51,676 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:01,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:01,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:57:01,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:01,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:01,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2527aeb6 >2014-10-23 10:57:01,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 2527aeb6 >2014-10-23 10:57:01,888 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:01,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:57:01,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:02,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 63453fc6 >2014-10-23 10:57:02,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 63453fc6 >2014-10-23 10:57:02,044 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:02,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:57:02,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:02,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6ae82af5 >2014-10-23 10:57:02,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 6ae82af5 >2014-10-23 10:57:02,171 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:12,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:12,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:57:12,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:12,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:12,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 9f69d47 >2014-10-23 10:57:12,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 9f69d47 >2014-10-23 10:57:12,344 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:12,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:57:12,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:12,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6223ceaa >2014-10-23 10:57:12,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 6223ceaa >2014-10-23 10:57:12,496 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:12,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:57:12,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:12,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6ff38d26 >2014-10-23 10:57:12,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 6ff38d26 >2014-10-23 10:57:12,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:22,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:22,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:57:22,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:22,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:22,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2523b546 >2014-10-23 10:57:22,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 2523b546 >2014-10-23 10:57:22,871 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:22,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:57:22,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:22,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 483ddded >2014-10-23 10:57:23,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 483ddded >2014-10-23 10:57:23,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:23,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:57:23,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:23,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3d246303 >2014-10-23 10:57:23,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 3d246303 >2014-10-23 10:57:23,143 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:33,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:33,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:57:33,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:33,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:33,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 50eab0c7 >2014-10-23 10:57:33,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 50eab0c7 >2014-10-23 10:57:33,354 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:33,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:57:33,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:33,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6183351f >2014-10-23 10:57:33,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 6183351f >2014-10-23 10:57:33,481 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:33,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:57:33,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:33,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 640a8c80 >2014-10-23 10:57:33,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 640a8c80 >2014-10-23 10:57:33,632 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:43,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:43,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:57:43,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:43,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:43,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 748a508a >2014-10-23 10:57:43,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 748a508a >2014-10-23 10:57:43,860 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:43,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:57:43,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:43,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7fcd7014 >2014-10-23 10:57:43,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 7fcd7014 >2014-10-23 10:57:43,980 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:44,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:57:44,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:44,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4fa78bea >2014-10-23 10:57:44,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 4fa78bea >2014-10-23 10:57:44,132 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:54,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:57:54,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:57:54,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:57:54,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:54,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: fd97692 >2014-10-23 10:57:54,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: fd97692 >2014-10-23 10:57:54,340 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:54,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:57:54,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:54,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6d0fcd12 >2014-10-23 10:57:54,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 6d0fcd12 >2014-10-23 10:57:54,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:57:54,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:57:54,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:57:54,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 32d1616b >2014-10-23 10:57:54,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 32d1616b >2014-10-23 10:57:54,612 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:04,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:04,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:58:04,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:04,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:04,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3d53eb86 >2014-10-23 10:58:04,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: 3d53eb86 >2014-10-23 10:58:04,782 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:04,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:58:04,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:04,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5b2522ec >2014-10-23 10:58:04,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: 5b2522ec >2014-10-23 10:58:04,926 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:04,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:58:05,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:05,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 409dd308 >2014-10-23 10:58:05,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: 409dd308 >2014-10-23 10:58:05,074 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:15,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:15,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:58:15,169 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:15,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:15,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5894a144 >2014-10-23 10:58:15,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 5894a144 >2014-10-23 10:58:15,289 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:15,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:58:15,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:15,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 790f6cdb >2014-10-23 10:58:15,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 790f6cdb >2014-10-23 10:58:15,448 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:15,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:58:15,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:15,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2f9396db >2014-10-23 10:58:15,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 2f9396db >2014-10-23 10:58:15,576 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:25,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:25,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:58:25,661 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:25,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:25,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 70dbe3f7 >2014-10-23 10:58:25,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, ConnectStoragePoolVDSCommand, log id: 70dbe3f7 >2014-10-23 10:58:25,750 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-18) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:25,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:58:25,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:25,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 323ca291 >2014-10-23 10:58:25,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, ConnectStoragePoolVDSCommand, log id: 323ca291 >2014-10-23 10:58:25,941 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-18) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:25,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:58:26,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:26,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3ac9e8e6 >2014-10-23 10:58:26,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, ConnectStoragePoolVDSCommand, log id: 3ac9e8e6 >2014-10-23 10:58:26,097 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-18) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:36,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:36,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:58:36,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:36,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:36,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2652d485 >2014-10-23 10:58:36,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 2652d485 >2014-10-23 10:58:36,351 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:36,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:58:36,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:36,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: aca9cf9 >2014-10-23 10:58:36,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: aca9cf9 >2014-10-23 10:58:36,483 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:36,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:58:36,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:36,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3992b382 >2014-10-23 10:58:36,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 3992b382 >2014-10-23 10:58:36,646 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:46,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:46,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:58:46,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:46,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:46,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5f3ad511 >2014-10-23 10:58:46,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 5f3ad511 >2014-10-23 10:58:46,836 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:46,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:58:46,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:46,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 70958e1 >2014-10-23 10:58:46,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 70958e1 >2014-10-23 10:58:46,991 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:47,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:58:47,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:47,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2914f16b >2014-10-23 10:58:47,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 2914f16b >2014-10-23 10:58:47,145 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:55,115 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-1) [34e0b354] Lock Acquired to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 10:58:55,116 WARN [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-1) [34e0b354] CanDoAction of action RemoveDisk failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_OVF_DISK_NOT_IN_APPLICABLE_STATUS,$status ILLEGAL >2014-10-23 10:58:55,116 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-1) [34e0b354] Lock freed to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 10:58:57,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:58:57,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:58:57,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:58:57,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:57,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: e2177a6 >2014-10-23 10:58:57,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: e2177a6 >2014-10-23 10:58:57,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:57,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:58:57,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:57,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4e84ef3 >2014-10-23 10:58:57,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 4e84ef3 >2014-10-23 10:58:57,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:58:57,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:58:57,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:58:57,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 16ab8a1e >2014-10-23 10:58:57,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 16ab8a1e >2014-10-23 10:58:57,754 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:07,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:59:07,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:59:07,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:59:07,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:07,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3b55749c >2014-10-23 10:59:07,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, ConnectStoragePoolVDSCommand, log id: 3b55749c >2014-10-23 10:59:07,992 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:08,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:59:08,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:08,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 38d0cdec >2014-10-23 10:59:08,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, ConnectStoragePoolVDSCommand, log id: 38d0cdec >2014-10-23 10:59:08,208 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:08,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:59:08,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:08,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 32e02de9 >2014-10-23 10:59:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, ConnectStoragePoolVDSCommand, log id: 32e02de9 >2014-10-23 10:59:08,332 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:18,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:59:18,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) HostName = monique-vds05.tlv.redhat.com >2014-10-23 10:59:18,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:59:18,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:18,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3c8d083a >2014-10-23 10:59:18,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, ConnectStoragePoolVDSCommand, log id: 3c8d083a >2014-10-23 10:59:18,512 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:18,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:59:18,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:18,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 35ff60fd >2014-10-23 10:59:18,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, ConnectStoragePoolVDSCommand, log id: 35ff60fd >2014-10-23 10:59:18,662 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:18,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:59:18,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:18,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1c9c0aa4 >2014-10-23 10:59:18,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, ConnectStoragePoolVDSCommand, log id: 1c9c0aa4 >2014-10-23 10:59:18,808 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:28,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:59:28,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 10:59:28,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:59:28,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:28,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4fdd292e >2014-10-23 10:59:29,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 4fdd292e >2014-10-23 10:59:29,013 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:29,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:59:29,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:29,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2d4cb464 >2014-10-23 10:59:29,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 2d4cb464 >2014-10-23 10:59:29,159 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:29,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:59:29,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:29,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 17f7d13e >2014-10-23 10:59:29,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 17f7d13e >2014-10-23 10:59:29,318 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:39,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:59:39,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:59:39,412 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:59:39,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:39,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3a3c2d88 >2014-10-23 10:59:39,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, ConnectStoragePoolVDSCommand, log id: 3a3c2d88 >2014-10-23 10:59:39,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:39,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 10:59:39,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:39,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2d4d885 >2014-10-23 10:59:39,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, ConnectStoragePoolVDSCommand, log id: 2d4d885 >2014-10-23 10:59:39,651 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:39,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:59:39,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:39,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 41d2dde0 >2014-10-23 10:59:39,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, ConnectStoragePoolVDSCommand, log id: 41d2dde0 >2014-10-23 10:59:39,795 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:49,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 10:59:49,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-50) HostName = monique-vds01.tlv.redhat.com >2014-10-23 10:59:49,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 10:59:49,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:49,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 52636447 >2014-10-23 10:59:50,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, ConnectStoragePoolVDSCommand, log id: 52636447 >2014-10-23 10:59:50,021 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:50,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 10:59:50,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:50,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 47b3b5bb >2014-10-23 10:59:50,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, ConnectStoragePoolVDSCommand, log id: 47b3b5bb >2014-10-23 10:59:50,172 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 10:59:50,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 10:59:50,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 10:59:50,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 336de82d >2014-10-23 10:59:50,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, ConnectStoragePoolVDSCommand, log id: 336de82d >2014-10-23 10:59:50,295 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-39) Autorecovering 1 storage domains >2014-10-23 11:00:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-39) Autorecovering storage domains id: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 >2014-10-23 11:00:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-39) [1b40409b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 Type: Storage >2014-10-23 11:00:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-39) [1b40409b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:10/23/14 11:00 AM >2014-10-23 11:00:00,052 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-37) [664d6cd4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:00:00,057 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-15) [55988e2a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:00:00,060 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-29) [10bc81f8] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:00:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [10bc81f8] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a04ed24 >2014-10-23 11:00:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [664d6cd4] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 25db70b9 >2014-10-23 11:00:00,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [10bc81f8] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: a04ed24 >2014-10-23 11:00:00,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [664d6cd4] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 25db70b9 >2014-10-23 11:00:00,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-15) [55988e2a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5f8dadf9 >2014-10-23 11:00:00,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-15) [55988e2a] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 5f8dadf9 >2014-10-23 11:00:00,154 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-39) [1b40409b] ConnectDomainToStorage. After Connect all hosts to pool. Time:10/23/14 11:00 AM >2014-10-23 11:00:00,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:00,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:00:00,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-95) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:00,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:00,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 367dce96 >2014-10-23 11:00:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 367dce96 >2014-10-23 11:00:00,477 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:00,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:00:00,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:00,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2a63e3c3 >2014-10-23 11:00:00,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 2a63e3c3 >2014-10-23 11:00:00,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:00,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:00:00,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 37575253 >2014-10-23 11:00:00,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 37575253 >2014-10-23 11:00:00,782 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:10,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:10,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:00:10,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:10,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:10,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 193e6ad0 >2014-10-23 11:00:11,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, ConnectStoragePoolVDSCommand, log id: 193e6ad0 >2014-10-23 11:00:11,006 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:11,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:00:11,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:11,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3f04f2c >2014-10-23 11:00:11,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3f04f2c >2014-10-23 11:00:11,212 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:11,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:00:11,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:11,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 72789acc >2014-10-23 11:00:11,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, ConnectStoragePoolVDSCommand, log id: 72789acc >2014-10-23 11:00:11,365 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:18,554 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [399d42c9] Lock Acquired to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 11:00:18,556 WARN [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [399d42c9] CanDoAction of action RemoveDisk failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status Inactive >2014-10-23 11:00:18,557 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [399d42c9] Lock freed to object EngineLock [exclusiveLocks= key: d97ea011-3053-4cb4-bbb7-9895b526ac22 value: DISK >, sharedLocks= ] >2014-10-23 11:00:19,795 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-10-23 11:00:20,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-10-23 11:00:21,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:21,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:00:21,570 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:21,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:21,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 153dffff >2014-10-23 11:00:21,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, ConnectStoragePoolVDSCommand, log id: 153dffff >2014-10-23 11:00:21,691 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:21,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:00:21,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:21,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 68e3fc5c >2014-10-23 11:00:21,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, ConnectStoragePoolVDSCommand, log id: 68e3fc5c >2014-10-23 11:00:21,854 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:21,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:00:21,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:21,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 69e64713 >2014-10-23 11:00:21,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, ConnectStoragePoolVDSCommand, log id: 69e64713 >2014-10-23 11:00:21,996 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:32,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:32,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:00:32,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:32,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:32,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4d2c28f6 >2014-10-23 11:00:32,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4d2c28f6 >2014-10-23 11:00:32,198 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:32,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:00:32,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:32,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 74f838fb >2014-10-23 11:00:32,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 74f838fb >2014-10-23 11:00:32,332 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:32,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:00:32,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:32,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 255f1d1a >2014-10-23 11:00:32,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 255f1d1a >2014-10-23 11:00:32,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:37,543 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (ajp-/127.0.0.1:8702-3) [3fea3717] Lock Acquired to object EngineLock [exclusiveLocks= key: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 value: STORAGE >, sharedLocks= ] >2014-10-23 11:00:37,988 WARN [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (ajp-/127.0.0.1:8702-3) [3fea3717] CanDoAction of action DetachStorageDomainFromPool failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DETACH,ERROR_CANNOT_DETACH_LAST_STORAGE_DOMAIN >2014-10-23 11:00:37,988 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (ajp-/127.0.0.1:8702-3) [3fea3717] Lock freed to object EngineLock [exclusiveLocks= key: f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 value: STORAGE >, sharedLocks= ] >2014-10-23 11:00:42,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:42,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:00:42,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:42,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:42,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1563da4b >2014-10-23 11:00:42,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, ConnectStoragePoolVDSCommand, log id: 1563da4b >2014-10-23 11:00:42,802 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:42,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:00:42,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:42,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7c3396db >2014-10-23 11:00:42,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, ConnectStoragePoolVDSCommand, log id: 7c3396db >2014-10-23 11:00:42,962 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:43,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:00:43,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:43,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5502a218 >2014-10-23 11:00:43,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, ConnectStoragePoolVDSCommand, log id: 5502a218 >2014-10-23 11:00:43,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:53,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:00:53,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:00:53,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:00:53,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:53,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 71a96322 >2014-10-23 11:00:53,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 71a96322 >2014-10-23 11:00:53,308 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:53,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:00:53,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:53,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3413ab09 >2014-10-23 11:00:53,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 3413ab09 >2014-10-23 11:00:53,457 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:00:53,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:00:53,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:00:53,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1a8c7f6f >2014-10-23 11:00:53,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] FINISH, ConnectStoragePoolVDSCommand, log id: 1a8c7f6f >2014-10-23 11:00:53,604 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6ee60cdb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:03,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:03,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:01:03,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:03,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:03,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 213f99f5 >2014-10-23 11:01:03,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 213f99f5 >2014-10-23 11:01:03,786 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:03,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:01:03,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:03,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1fbd89f4 >2014-10-23 11:01:03,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 1fbd89f4 >2014-10-23 11:01:03,933 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:03,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:01:04,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:04,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1037a08 >2014-10-23 11:01:04,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 1037a08 >2014-10-23 11:01:04,108 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:14,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:14,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:01:14,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:14,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:14,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 39c4ba70 >2014-10-23 11:01:14,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 39c4ba70 >2014-10-23 11:01:14,328 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:14,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:01:14,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:14,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2a8cfe95 >2014-10-23 11:01:14,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 2a8cfe95 >2014-10-23 11:01:14,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:14,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:01:14,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:14,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2b8b1d3e >2014-10-23 11:01:14,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 2b8b1d3e >2014-10-23 11:01:14,649 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:24,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:24,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:01:24,763 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:24,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:24,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5fe33982 >2014-10-23 11:01:24,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 5fe33982 >2014-10-23 11:01:24,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:24,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:01:24,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:25,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 5f3dad0c >2014-10-23 11:01:25,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 5f3dad0c >2014-10-23 11:01:25,061 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:25,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:01:25,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:25,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 7ee77f1e >2014-10-23 11:01:25,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 7ee77f1e >2014-10-23 11:01:25,192 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:35,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:35,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:01:35,299 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:35,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:35,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4b9b5990 >2014-10-23 11:01:35,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 4b9b5990 >2014-10-23 11:01:35,426 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:35,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:01:35,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:35,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6f82dc52 >2014-10-23 11:01:35,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 6f82dc52 >2014-10-23 11:01:35,546 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:35,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:01:35,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:35,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 689fde14 >2014-10-23 11:01:35,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 689fde14 >2014-10-23 11:01:35,698 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:45,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:45,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:01:45,794 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:45,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:45,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 6fae3ede >2014-10-23 11:01:45,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6fae3ede >2014-10-23 11:01:45,892 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:45,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:01:45,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:46,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 49432cc0 >2014-10-23 11:01:46,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 49432cc0 >2014-10-23 11:01:46,038 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:46,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:01:46,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:46,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1dc69f85 >2014-10-23 11:01:46,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1dc69f85 >2014-10-23 11:01:46,190 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:50,378 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [157f52de] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:01:50,418 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [157f52de] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:01:50,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [157f52de] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e422fb7 >2014-10-23 11:01:50,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [157f52de] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6e422fb7 >2014-10-23 11:01:51,047 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [157f52de] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:01:51,425 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:01:51,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: fc398c1 >2014-10-23 11:01:51,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] FINISH, ConnectStorageServerVDSCommand, return: {2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e=0}, log id: fc398c1 >2014-10-23 11:01:51,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] START, CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storageDomain=StorageDomainStatic[SD2, cc5bf5fb-d743-4682-9ec6-35c182131409], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 26dc0531 >2014-10-23 11:01:51,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Failed in CreateStorageDomainVDS method >2014-10-23 11:01:51,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=cc5bf5fb-d743-4682-9ec6-35c182131409, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',)]] >2014-10-23 11:01:51,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:01:51,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Command CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storageDomain=StorageDomainStatic[SD2, cc5bf5fb-d743-4682-9ec6-35c182131409], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=cc5bf5fb-d743-4682-9ec6-35c182131409, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 >2014-10-23 11:01:51,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] FINISH, CreateStorageDomainVDSCommand, log id: 26dc0531 >2014-10-23 11:01:51,558 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=cc5bf5fb-d743-4682-9ec6-35c182131409, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:01:51,566 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Command [id=3c9cf97a-e671-44d6-8f2b-68dbd0b3123b]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: cc5bf5fb-d743-4682-9ec6-35c182131409. >2014-10-23 11:01:51,577 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Command [id=3c9cf97a-e671-44d6-8f2b-68dbd0b3123b]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: cc5bf5fb-d743-4682-9ec6-35c182131409. >2014-10-23 11:01:51,662 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [665b94a4] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:01:51,732 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [665b94a4] Correlation ID: 665b94a4, Job ID: 933dabc7-b4ee-4895-9397-ea3228e6a9d5, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain SD2. (User: admin) >2014-10-23 11:01:52,068 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:01:52,130 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:01:52,186 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Removing connection 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e from database >2014-10-23 11:01:52,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 51595e87 >2014-10-23 11:01:52,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Failed in DisconnectStorageServerVDS method >2014-10-23 11:01:52,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e', u'port': u''}]",)]] > >2014-10-23 11:01:52,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:01:52,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Command DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e', u'port': u''}]",), code = 452 >2014-10-23 11:01:52,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5692d678] FINISH, DisconnectStorageServerVDSCommand, log id: 51595e87 >2014-10-23 11:01:52,265 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:01:52,336 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [5692d678] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: 2a44a5f6-b4cb-43ac-9d1e-a49499f8dc2e value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:01:56,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:01:56,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:01:56,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:01:56,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:56,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 66c8edda >2014-10-23 11:01:56,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, ConnectStoragePoolVDSCommand, log id: 66c8edda >2014-10-23 11:01:56,443 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:56,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:01:56,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:56,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 55584a9b >2014-10-23 11:01:56,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, ConnectStoragePoolVDSCommand, log id: 55584a9b >2014-10-23 11:01:56,617 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:01:56,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:01:56,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:01:56,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 19c9382d >2014-10-23 11:01:56,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, ConnectStoragePoolVDSCommand, log id: 19c9382d >2014-10-23 11:01:56,810 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:00,785 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [f07f68c] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:02:00,834 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [f07f68c] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [f07f68c] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 257e0775 >2014-10-23 11:02:00,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [f07f68c] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 257e0775 >2014-10-23 11:02:00,911 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [f07f68c] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:02:01,079 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:01,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ad5a3079-a40a-4419-b926-6ab8fdda37c4, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6354d0aa >2014-10-23 11:02:01,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] FINISH, ConnectStorageServerVDSCommand, return: {ad5a3079-a40a-4419-b926-6ab8fdda37c4=0}, log id: 6354d0aa >2014-10-23 11:02:01,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] START, CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storageDomain=StorageDomainStatic[SD2, bb42a94b-e391-42c3-a329-bf40e63b7db7], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 67363f37 >2014-10-23 11:02:01,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Failed in CreateStorageDomainVDS method >2014-10-23 11:02:01,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=bb42a94b-e391-42c3-a329-bf40e63b7db7, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',)]] >2014-10-23 11:02:01,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:02:01,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Command CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storageDomain=StorageDomainStatic[SD2, bb42a94b-e391-42c3-a329-bf40e63b7db7], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=bb42a94b-e391-42c3-a329-bf40e63b7db7, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 >2014-10-23 11:02:01,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] FINISH, CreateStorageDomainVDSCommand, log id: 67363f37 >2014-10-23 11:02:01,163 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=bb42a94b-e391-42c3-a329-bf40e63b7db7, domainName=SD2, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:02:01,166 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Command [id=7f0b7a97-fb7c-4123-a469-543ac7a4ad17]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: bb42a94b-e391-42c3-a329-bf40e63b7db7. >2014-10-23 11:02:01,167 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Command [id=7f0b7a97-fb7c-4123-a469-543ac7a4ad17]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: bb42a94b-e391-42c3-a329-bf40e63b7db7. >2014-10-23 11:02:01,180 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [26181bf9] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:02:01,185 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [26181bf9] Correlation ID: 26181bf9, Job ID: 8b5366a0-8ba6-4c72-97fb-8531ace40679, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain SD2. (User: admin) >2014-10-23 11:02:01,459 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: ad5a3079-a40a-4419-b926-6ab8fdda37c4 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:02:01,555 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:01,559 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Removing connection ad5a3079-a40a-4419-b926-6ab8fdda37c4 from database >2014-10-23 11:02:01,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ad5a3079-a40a-4419-b926-6ab8fdda37c4, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7f386af4 >2014-10-23 11:02:01,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Failed in DisconnectStorageServerVDS method >2014-10-23 11:02:01,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'ad5a3079-a40a-4419-b926-6ab8fdda37c4', u'port': u''}]",)]] > >2014-10-23 11:02:01,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:02:01,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Command DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ad5a3079-a40a-4419-b926-6ab8fdda37c4, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'ad5a3079-a40a-4419-b926-6ab8fdda37c4', u'port': u''}]",), code = 452 >2014-10-23 11:02:01,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [baa309e] FINISH, DisconnectStorageServerVDSCommand, log id: 7f386af4 >2014-10-23 11:02:01,631 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'ad5a3079-a40a-4419-b926-6ab8fdda37c4', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:02:01,637 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [baa309e] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: ad5a3079-a40a-4419-b926-6ab8fdda37c4 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:02:06,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:02:06,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:02:06,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:02:07,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:07,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3194c80d >2014-10-23 11:02:07,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 3194c80d >2014-10-23 11:02:07,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:07,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:02:07,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:07,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 71dc6dfe >2014-10-23 11:02:07,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 71dc6dfe >2014-10-23 11:02:07,219 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:07,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:02:07,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:07,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4fee94a0 >2014-10-23 11:02:07,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 4fee94a0 >2014-10-23 11:02:07,469 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:09,655 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePoolAction group DELETE_STORAGE_POOL with role type ADMIN >2014-10-23 11:02:10,076 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-36) [5744f27a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:10,081 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-46) [12dc3526] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:10,139 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-25) [50cdbf3d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:02:10,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-46) [12dc3526] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5026c4d9 >2014-10-23 11:02:10,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-25) [50cdbf3d] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 27c000e3 >2014-10-23 11:02:10,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-46) [12dc3526] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 5026c4d9 >2014-10-23 11:02:10,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-46) [12dc3526] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 30574071 >2014-10-23 11:02:10,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-25) [50cdbf3d] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 27c000e3 >2014-10-23 11:02:10,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-25) [50cdbf3d] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 19d83db8 >2014-10-23 11:02:10,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-25) [50cdbf3d] FINISH, ConnectStoragePoolVDSCommand, log id: 19d83db8 >2014-10-23 11:02:10,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-46) [12dc3526] FINISH, ConnectStoragePoolVDSCommand, log id: 30574071 >2014-10-23 11:02:10,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-36) [5744f27a] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f514a63 >2014-10-23 11:02:10,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-36) [5744f27a] FINISH, ConnectStorageServerVDSCommand, return: {33a7c725-8f47-4695-a869-e05bcc3ac716=0}, log id: 3f514a63 >2014-10-23 11:02:10,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-36) [5744f27a] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 3837a137 >2014-10-23 11:02:10,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-36) [5744f27a] FINISH, ConnectStoragePoolVDSCommand, log id: 3837a137 >2014-10-23 11:02:11,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, DestroyStoragePoolVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false), log id: 4eec3e62 >2014-10-23 11:02:11,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:11,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 31285a6b >2014-10-23 11:02:11,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 31285a6b >2014-10-23 11:02:11,291 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::DestroyStoragePoolVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:11,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:02:11,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:11,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2033004f >2014-10-23 11:02:11,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 2033004f >2014-10-23 11:02:11,553 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::DestroyStoragePoolVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:11,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:02:11,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:11,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 25d2b87a >2014-10-23 11:02:11,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 25d2b87a >2014-10-23 11:02:11,704 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::DestroyStoragePoolVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:11,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, DestroyStoragePoolVDSCommand, log id: 4eec3e62 >2014-10-23 11:02:11,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, SpmStopOnIrsVDSCommand( storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, ignoreFailoverLimit = false), log id: 361bc8b3 >2014-10-23 11:02:11,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:11,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 25f0f00f >2014-10-23 11:02:11,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 25f0f00f >2014-10-23 11:02:11,867 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::SpmStopOnIrsVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:11,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] Irs placed on server e9342ea1-ce56-435b-8284-0e4febfec3c1 failed. Proceed Failover >2014-10-23 11:02:11,959 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:11,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 68c24012 >2014-10-23 11:02:12,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 68c24012 >2014-10-23 11:02:12,022 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::SpmStopOnIrsVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:12,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:02:12,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-6) [644749c3] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:12,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 67d4f940 >2014-10-23 11:02:12,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, ConnectStoragePoolVDSCommand, log id: 67d4f940 >2014-10-23 11:02:12,211 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] IrsBroker::Failed::SpmStopOnIrsVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:12,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] FINISH, SpmStopOnIrsVDSCommand, log id: 361bc8b3 >2014-10-23 11:02:12,211 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (org.ovirt.thread.pool-7-thread-6) [644749c3] transaction rolled back >2014-10-23 11:02:12,248 ERROR [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Failed destroy storage pool with id 1616f9cd-9b21-400e-b38b-2354bf808db8 and after that failed to stop spm because of org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2031) [bll.jar:] > at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:639) [bll.jar:] > at org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand$7.runInTransaction(RemoveStoragePoolCommand.java:238) [bll.jar:] > at org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand$7.runInTransaction(RemoveStoragePoolCommand.java:235) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:210) [utils.jar:] > at org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand.handleDestroyStoragePoolCommand(RemoveStoragePoolCommand.java:235) [bll.jar:] > at org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand.regularRemoveStorageDomains(RemoveStoragePoolCommand.java:203) [bll.jar:] > at org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand.executeCommand(RemoveStoragePoolCommand.java:76) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:191) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:158) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:167) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] > >2014-10-23 11:02:12,249 ERROR [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Command org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001) >2014-10-23 11:02:12,345 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Command [id=1ce70d88-9536-4d94-9354-6b8706a44609]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=f548e584-2ca0-41da-9eb4-e5f6de3a4fc7. >2014-10-23 11:02:12,383 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Command [id=1ce70d88-9536-4d94-9354-6b8706a44609]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: id=storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageId = f548e584-2ca0-41da-9eb4-e5f6de3a4fc7. >2014-10-23 11:02:12,540 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Command [id=1ce70d88-9536-4d94-9354-6b8706a44609]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.network.Network; snapshot: id=6c46e0dd-be51-44cb-98ae-41fc21298712. >2014-10-23 11:02:12,613 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (org.ovirt.thread.pool-7-thread-6) [644749c3] Command [id=1ce70d88-9536-4d94-9354-6b8706a44609]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.network.VnicProfile; snapshot: id=3b8390f7-d8b5-469d-afc7-bdc780ea8f97. >2014-10-23 11:02:12,745 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-6) [644749c3] Correlation ID: 644749c3, Job ID: 8d96e415-0816-4c77-8d0f-316f66f89b17, Call Stack: null, Custom Event ID: -1, Message: Failed to remove Data Center DC1. (User: admin) >2014-10-23 11:02:17,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:02:17,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:02:17,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:02:17,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:17,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 1b67597b >2014-10-23 11:02:17,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] FINISH, ConnectStoragePoolVDSCommand, log id: 1b67597b >2014-10-23 11:02:17,828 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:17,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:02:17,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:17,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 284ce32d >2014-10-23 11:02:18,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] FINISH, ConnectStoragePoolVDSCommand, log id: 284ce32d >2014-10-23 11:02:18,034 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:18,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:02:18,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:18,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 65911444 >2014-10-23 11:02:18,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] FINISH, ConnectStoragePoolVDSCommand, log id: 65911444 >2014-10-23 11:02:18,201 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [4f66a0c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:28,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:02:28,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:02:28,337 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:02:28,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:28,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 2b818b1a >2014-10-23 11:02:28,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, ConnectStoragePoolVDSCommand, log id: 2b818b1a >2014-10-23 11:02:28,514 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:28,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) Irs placed on server 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c failed. Proceed Failover >2014-10-23 11:02:28,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:28,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vdsId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 4b891d5e >2014-10-23 11:02:28,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, ConnectStoragePoolVDSCommand, log id: 4b891d5e >2014-10-23 11:02:28,668 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:28,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) Irs placed on server d8177efa-36f1-40fa-b2d6-01cbe63b3982 failed. Proceed Failover >2014-10-23 11:02:28,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:02:28,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vdsId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, masterVersion = 1), log id: 53e3bb92 >2014-10-23 11:02:28,803 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:02:28,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 3b02377d >2014-10-23 11:02:28,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, ConnectStoragePoolVDSCommand, log id: 53e3bb92 >2014-10-23 11:02:28,830 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) >2014-10-23 11:02:31,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] FINISH, SetVdsStatusVDSCommand, log id: 3b02377d >2014-10-23 11:02:32,020 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:02:32,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] START, SetHaMaintenanceModeVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 4ad37c23 >2014-10-23 11:02:32,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4ad37c23 >2014-10-23 11:02:32,037 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-9) [3e2bff72] Correlation ID: 3e2bff72, Job ID: 9e1d0e50-441c-447b-b2d0-b25b048410d3, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was switched to Maintenance mode by admin. >2014-10-23 11:02:32,628 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-20) [6d72957a] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:02:32,632 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-20) [6d72957a] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 7f37397b >2014-10-23 11:02:32,643 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-20) [6d72957a] FINISH, SetVdsStatusVDSCommand, log id: 7f37397b >2014-10-23 11:02:32,836 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-20) [6d72957a] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:02:32,851 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-20) [6d72957a] Correlation ID: 6d72957a, Job ID: 7cbff246-bf06-47aa-85e6-d11ce50b4aa5, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was switched to Maintenance mode by admin. >2014-10-23 11:02:33,072 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = e9342ea1-ce56-435b-8284-0e4febfec3c1 : monique-vds05.tlv.redhat.com >2014-10-23 11:02:33,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Clearing cache of pool: 1616f9cd-9b21-400e-b38b-2354bf808db8 for problematic entities of VDS: monique-vds05.tlv.redhat.com. >2014-10-23 11:02:33,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Removing vds [e9342ea1-ce56-435b-8284-0e4febfec3c1] from the domain in maintenance cache >2014-10-23 11:02:33,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Removing host(s) [e9342ea1-ce56-435b-8284-0e4febfec3c1] from hosts unseen domain report cache >2014-10-23 11:02:33,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, vds_spm_id = 3), log id: 77eb6810 >2014-10-23 11:02:33,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, DisconnectStoragePoolVDSCommand, log id: 77eb6810 >2014-10-23 11:02:33,142 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 11:02:33,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6abce600 >2014-10-23 11:02:33,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] Failed in DisconnectStorageServerVDS method >2014-10-23 11:02:33,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",)]] > >2014-10-23 11:02:33,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:02:33,218 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] Command DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 >2014-10-23 11:02:33,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] FINISH, DisconnectStorageServerVDSCommand, log id: 6abce600 >2014-10-23 11:02:33,218 ERROR [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [7128d744] Command org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:02:36,973 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:02:37,058 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 5aafb725 >2014-10-23 11:02:37,088 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] FINISH, SetVdsStatusVDSCommand, log id: 5aafb725 >2014-10-23 11:02:37,407 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:02:37,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 286e2403 >2014-10-23 11:02:37,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 286e2403 >2014-10-23 11:02:37,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-18) [5d44b2e6] Correlation ID: 5d44b2e6, Job ID: b0f8f6f9-d2bb-4891-8bff-ba82cbc4a4fe, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was switched to Maintenance mode by admin. >2014-10-23 11:02:38,954 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-97) Storage Pool 1616f9cd-9b21-400e-b38b-2354bf808db8 - Updating Storage Domain f548e584-2ca0-41da-9eb4-e5f6de3a4fc7 status from Inactive to Unknown, reason : no reporting hosts >2014-10-23 11:02:38,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-10-23 11:02:38,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) HostName = monique-vds05.tlv.redhat.com >2014-10-23 11:02:38,989 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command HSMGetAllTasksStatusesVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-10-23 11:02:50,577 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-8) [65179983] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 11:02:50,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [65179983] Correlation ID: 65179983, Call Stack: null, Custom Event ID: -1, Message: Data Center DC11, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-10-23 11:02:59,930 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = d8177efa-36f1-40fa-b2d6-01cbe63b3982 : monique-vds01.tlv.redhat.com >2014-10-23 11:03:00,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-17) Clearing cache of pool: 1616f9cd-9b21-400e-b38b-2354bf808db8 for problematic entities of VDS: monique-vds01.tlv.redhat.com. >2014-10-23 11:03:00,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-17) Removing vds [d8177efa-36f1-40fa-b2d6-01cbe63b3982] from the domain in maintenance cache >2014-10-23 11:03:00,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-17) Removing host(s) [d8177efa-36f1-40fa-b2d6-01cbe63b3982] from hosts unseen domain report cache >2014-10-23 11:03:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, vds_spm_id = 2), log id: 6bac84c7 >2014-10-23 11:03:00,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, DisconnectStoragePoolVDSCommand, log id: 6bac84c7 >2014-10-23 11:03:00,068 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 11:03:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a1767f5 >2014-10-23 11:03:00,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] Failed in DisconnectStorageServerVDS method >2014-10-23 11:03:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",)]] > >2014-10-23 11:03:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:03:00,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 >2014-10-23 11:03:00,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] FINISH, DisconnectStorageServerVDSCommand, log id: a1767f5 >2014-10-23 11:03:00,195 ERROR [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [7ffa2700] Command org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:03:04,028 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com >2014-10-23 11:03:04,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-3) Clearing cache of pool: 1616f9cd-9b21-400e-b38b-2354bf808db8 for problematic entities of VDS: rose07.qa.lab.tlv.redhat.com. >2014-10-23 11:03:04,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-3) Removing vds [3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c] from the domain in maintenance cache >2014-10-23 11:03:04,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-3) Removing host(s) [3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c] from hosts unseen domain report cache >2014-10-23 11:03:04,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-22) START, DisconnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, vds_spm_id = 1), log id: 3589b8cc >2014-10-23 11:03:04,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, DisconnectStoragePoolVDSCommand, log id: 3589b8cc >2014-10-23 11:03:04,092 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePool >2014-10-23 11:03:04,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] START, DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70f8db23 >2014-10-23 11:03:04,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] Failed in DisconnectStorageServerVDS method >2014-10-23 11:03:04,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",)]] > >2014-10-23 11:03:04,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 11:03:04,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] Command DisconnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, storagePoolId = 1616f9cd-9b21-400e-b38b-2354bf808db8, storageType = NFS, connectionList = [{ id: 33a7c725-8f47-4695-a869-e05bcc3ac716, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 >2014-10-23 11:03:04,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] FINISH, DisconnectStorageServerVDSCommand, log id: 70f8db23 >2014-10-23 11:03:04,252 ERROR [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [360c030a] Command org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=1616f9cd-9b21-400e-b38b-2354bf808db8, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33a7c725-8f47-4695-a869-e05bcc3ac716', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:03:35,963 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-2) [d2cf7f3] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 Type: StoragePoolAction group CREATE_CLUSTER with role type ADMIN >2014-10-23 11:03:36,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [d2cf7f3] Correlation ID: d2cf7f3, Call Stack: null, Custom Event ID: -1, Message: Cluster cluster11 was added by admin >2014-10-23 11:03:44,307 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-7-thread-27) [6125b675] Start running CanDoAction for command number 1/2 (Command type: ChangeVDSCluster) >2014-10-23 11:03:44,308 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-7-thread-33) [72eded26] Start running CanDoAction for command number 2/2 (Command type: ChangeVDSCluster) >2014-10-23 11:03:44,333 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-27) [6125b675] Lock Acquired to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 11:03:44,362 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-7-thread-27) [6125b675] Finish handling CanDoAction for command number 1/2 (Command type: ChangeVDSCluster) >2014-10-23 11:03:44,395 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-33) [72eded26] Lock Acquired to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 11:03:44,400 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-7-thread-33) [72eded26] Finish handling CanDoAction for command number 2/2 (Command type: ChangeVDSCluster) >2014-10-23 11:03:44,401 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN, ID: a26c582a-cf59-4411-9c4f-e5a152b7127d Type: VdsGroupsAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 11:03:44,465 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] START, RemoveVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1), log id: 600af847 >2014-10-23 11:03:44,465 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [6125b675] vdsManager::disposing >2014-10-23 11:03:44,469 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] FINISH, RemoveVdsVDSCommand, log id: 600af847 >2014-10-23 11:03:44,562 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] START, AddVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1), log id: 77c90fa7 >2014-10-23 11:03:44,563 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] AddVds - entered , starting logic to add VDS e9342ea1-ce56-435b-8284-0e4febfec3c1 >2014-10-23 11:03:44,565 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] AddVds - VDS e9342ea1-ce56-435b-8284-0e4febfec3c1 was added, will try to add it to the resource manager >2014-10-23 11:03:44,565 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [6125b675] Entered VdsManager constructor >2014-10-23 11:03:44,699 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [6125b675] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-23 11:03:44,722 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (org.ovirt.thread.pool-7-thread-47) [6125b675] VDS e9342ea1-ce56-435b-8284-0e4febfec3c1 was added to the Resource Manager >2014-10-23 11:03:44,722 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [6125b675] FINISH, AddVdsVDSCommand, log id: 77c90fa7 >2014-10-23 11:03:44,748 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-47) [9eb4ed0] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:03:44,771 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-47) [9eb4ed0] Lock freed to object EngineLock [exclusiveLocks= key: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 11:03:44,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-47) [9eb4ed0] Correlation ID: 6125b675, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-10-23 11:03:44,852 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-47) [9eb4ed0] Lock freed to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 11:03:44,852 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN, ID: a26c582a-cf59-4411-9c4f-e5a152b7127d Type: VdsGroupsAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 11:03:44,962 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-7-thread-11) [23262ac3] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:03:45,021 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] START, RemoveVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 4ce6dcca >2014-10-23 11:03:45,021 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [72eded26] vdsManager::disposing >2014-10-23 11:03:45,106 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-11) [23262ac3] Running command: SetupNetworksCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-10-23 11:03:45,106 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-11) [23262ac3] No changes were detected in setup networks for host monique-vds05.tlv.redhat.com (ID: e9342ea1-ce56-435b-8284-0e4febfec3c1) >2014-10-23 11:03:45,107 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] FINISH, RemoveVdsVDSCommand, log id: 4ce6dcca >2014-10-23 11:03:45,167 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] START, AddVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 53e03a94 >2014-10-23 11:03:45,168 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] AddVds - entered , starting logic to add VDS 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c >2014-10-23 11:03:45,285 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] AddVds - VDS 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c was added, will try to add it to the resource manager >2014-10-23 11:03:45,285 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [72eded26] Entered VdsManager constructor >2014-10-23 11:03:45,374 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-47) [72eded26] Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-10-23 11:03:45,374 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (org.ovirt.thread.pool-7-thread-47) [72eded26] VDS 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c was added to the Resource Manager >2014-10-23 11:03:45,374 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-47) [72eded26] FINISH, AddVdsVDSCommand, log id: 53e03a94 >2014-10-23 11:03:45,476 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-47) [125851f0] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:03:45,567 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-47) [125851f0] Lock freed to object EngineLock [exclusiveLocks= key: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 11:03:46,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-47) [125851f0] Correlation ID: 72eded26, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-10-23 11:03:46,022 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-47) [125851f0] Lock freed to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 11:03:46,140 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-7-thread-40) [6ce888fc] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:03:46,361 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-40) [6ce888fc] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-10-23 11:03:46,364 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-40) [6ce888fc] No changes were detected in setup networks for host rose07.qa.lab.tlv.redhat.com (ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c) >2014-10-23 11:03:47,725 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-23 11:03:48,549 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [23d50e56] Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-10-23 11:04:05,074 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-9) [6961ad9c] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 Type: StoragePoolAction group CREATE_CLUSTER with role type ADMIN >2014-10-23 11:04:05,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [6961ad9c] Correlation ID: 6961ad9c, Call Stack: null, Custom Event ID: -1, Message: Cluster cluster12 was added by admin >2014-10-23 11:04:10,030 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-10) [b1feb4b] Lock Acquired to object EngineLock [exclusiveLocks= key: d8177efa-36f1-40fa-b2d6-01cbe63b3982 value: VDS >, sharedLocks= ] >2014-10-23 11:04:10,041 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN, ID: b22ac099-3faf-4682-b22b-9b85f3066bf8 Type: VdsGroupsAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 11:04:10,055 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] START, RemoveVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 67ef8180 >2014-10-23 11:04:10,056 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] vdsManager::disposing >2014-10-23 11:04:10,056 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] FINISH, RemoveVdsVDSCommand, log id: 67ef8180 >2014-10-23 11:04:10,058 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] START, AddVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 688c83c >2014-10-23 11:04:10,059 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] AddVds - entered , starting logic to add VDS d8177efa-36f1-40fa-b2d6-01cbe63b3982 >2014-10-23 11:04:10,060 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] AddVds - VDS d8177efa-36f1-40fa-b2d6-01cbe63b3982 was added, will try to add it to the resource manager >2014-10-23 11:04:10,060 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] Entered VdsManager constructor >2014-10-23 11:04:10,073 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-10-23 11:04:10,074 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] VDS d8177efa-36f1-40fa-b2d6-01cbe63b3982 was added to the Resource Manager >2014-10-23 11:04:10,074 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (org.ovirt.thread.pool-7-thread-49) [b1feb4b] FINISH, AddVdsVDSCommand, log id: 688c83c >2014-10-23 11:04:10,080 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-49) [4bc40793] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:04:10,091 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (org.ovirt.thread.pool-7-thread-49) [4bc40793] Lock freed to object EngineLock [exclusiveLocks= key: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 11:04:10,212 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-7-thread-29) [3dcee004] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:04:10,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-49) [4bc40793] Correlation ID: b1feb4b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com configuration was updated by admin. >2014-10-23 11:04:10,231 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (org.ovirt.thread.pool-7-thread-49) [4bc40793] Lock freed to object EngineLock [exclusiveLocks= key: d8177efa-36f1-40fa-b2d6-01cbe63b3982 value: VDS >, sharedLocks= ] >2014-10-23 11:04:10,359 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-29) [3dcee004] Running command: SetupNetworksCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-10-23 11:04:10,360 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-29) [3dcee004] No changes were detected in setup networks for host monique-vds01.tlv.redhat.com (ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982) >2014-10-23 11:04:13,082 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Initializing Host: monique-vds01.tlv.redhat.com >2014-10-23 11:04:17,989 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-6) [5eb85af7] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: a26c582a-cf59-4411-9c4f-e5a152b7127d Type: VdsGroupsAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN >2014-10-23 11:04:18,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [5eb85af7] Correlation ID: 5eb85af7, Call Stack: null, Custom Event ID: -1, Message: Host cluster cluster11 was updated by admin >2014-10-23 11:04:25,539 WARN [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-7) [34c1b8bf] CanDoAction of action RemoveStoragePool failed. Reasons:VAR__TYPE__STORAGE__POOL,VAR__ACTION__REMOVE,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-10-23 11:04:31,529 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [6f741002] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 1616f9cd-9b21-400e-b38b-2354bf808db8 Type: StoragePoolAction group DELETE_STORAGE_POOL with role type ADMIN >2014-10-23 11:04:31,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [6f741002] Correlation ID: 6f741002, Job ID: 575670ec-c0b2-4004-b87a-7b1252bde6f7, Call Stack: null, Custom Event ID: -1, Message: Data Center DC1 was forcibly removed by admin >2014-10-23 11:04:37,170 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (org.ovirt.thread.pool-7-thread-15) [5432be0e] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 25442e43-58f4-4a00-96ed-41d78a53384e Type: VdsGroupsAction group DELETE_CLUSTER with role type ADMIN >2014-10-23 11:04:37,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-15) [5432be0e] Correlation ID: 5432be0e, Call Stack: null, Custom Event ID: -1, Message: Host cluster cluster1 was removed by admin >2014-10-23 11:04:40,510 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (org.ovirt.thread.pool-7-thread-26) [75e17082] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: b7f5eb96-f77b-409d-866e-b50935978b58 Type: VdsGroupsAction group DELETE_CLUSTER with role type ADMIN >2014-10-23 11:04:40,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-26) [75e17082] Correlation ID: 75e17082, Call Stack: null, Custom Event ID: -1, Message: Host cluster cluster2 was removed by admin >2014-10-23 11:04:59,791 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) [3b45470f] Lock Acquired to object EngineLock [exclusiveLocks= key: d8177efa-36f1-40fa-b2d6-01cbe63b3982 value: VDS >, sharedLocks= ] >2014-10-23 11:04:59,822 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Running command: ActivateVdsCommand internal: false. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:04:59,823 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Before acquiring lock in order to prevent monitoring for host monique-vds01.tlv.redhat.com from data-center DC11 >2014-10-23 11:04:59,823 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Lock acquired, from now a monitoring of host will be skipped for host monique-vds01.tlv.redhat.com from data-center DC11 >2014-10-23 11:04:59,873 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: f7dec27 >2014-10-23 11:04:59,902 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] FINISH, SetVdsStatusVDSCommand, log id: f7dec27 >2014-10-23 11:04:59,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] START, SetHaMaintenanceModeVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 5f6fa05b >2014-10-23 11:04:59,957 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 11:05:00,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5f6fa05b >2014-10-23 11:05:00,140 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Activate finished. Lock released. Monitoring can run now for host monique-vds01.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:00,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Correlation ID: 3b45470f, Job ID: 21605cc2-b642-47e4-a896-19788c1f1a7b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was activated by admin. >2014-10-23 11:05:00,154 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-8) [3b45470f] Lock freed to object EngineLock [exclusiveLocks= key: d8177efa-36f1-40fa-b2d6-01cbe63b3982 value: VDS >, sharedLocks= ] >2014-10-23 11:05:01,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]), log id: 331995c8 >2014-10-23 11:05:01,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, GetHardwareInfoVDSCommand, log id: 331995c8 >2014-10-23 11:05:01,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [39fa8905] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:05:01,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [39fa8905] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]), log id: 6d0d8190 >2014-10-23 11:05:01,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [39fa8905] FINISH, GetHardwareInfoVDSCommand, log id: 6d0d8190 >2014-10-23 11:05:01,403 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [471baa21] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 Type: StoragePool >2014-10-23 11:05:01,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [471baa21] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982), log id: 57bab87c >2014-10-23 11:05:01,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [471baa21] FINISH, SetMOMPolicyParametersVDSCommand, log id: 57bab87c >2014-10-23 11:05:01,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [471baa21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-10-23 11:05:01,529 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [471baa21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-10-23 11:05:01,554 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [46f7abfd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:05:02,415 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-4) [5793e831] Lock Acquired to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 11:05:02,525 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] Running command: ActivateVdsCommand internal: false. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:05:02,525 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] Before acquiring lock in order to prevent monitoring for host monique-vds05.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:02,525 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] Lock acquired, from now a monitoring of host will be skipped for host monique-vds05.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:02,553 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5104418e >2014-10-23 11:05:02,564 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] FINISH, SetVdsStatusVDSCommand, log id: 5104418e >2014-10-23 11:05:02,812 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] Activate finished. Lock released. Monitoring can run now for host monique-vds05.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:02,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-39) [5793e831] Correlation ID: 5793e831, Job ID: fbf3d7d1-c5b1-4968-ad0c-43cbd63082fd, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was activated by admin. >2014-10-23 11:05:02,845 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-39) [5793e831] Lock freed to object EngineLock [exclusiveLocks= key: e9342ea1-ce56-435b-8284-0e4febfec3c1 value: VDS >, sharedLocks= ] >2014-10-23 11:05:03,579 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:05:03,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 2e6f9a5f >2014-10-23 11:05:03,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, GetHardwareInfoVDSCommand, log id: 2e6f9a5f >2014-10-23 11:05:03,781 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [405e5aad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:05:03,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [405e5aad] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 24ed4625 >2014-10-23 11:05:03,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [405e5aad] FINISH, GetHardwareInfoVDSCommand, log id: 24ed4625 >2014-10-23 11:05:03,950 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-99) [4f231eed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 Type: StoragePool >2014-10-23 11:05:03,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-99) [4f231eed] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1), log id: 2c4760e6 >2014-10-23 11:05:04,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-99) [4f231eed] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2c4760e6 >2014-10-23 11:05:04,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4f231eed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-10-23 11:05:04,052 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4f231eed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 11:05:04,122 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [4c327c12] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:05:06,593 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-3) [78e875d] Lock Acquired to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 11:05:06,672 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-25) [78e875d] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 11:05:06,672 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-25) [78e875d] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:06,672 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-25) [78e875d] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:06,772 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [78e875d] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d36f782 >2014-10-23 11:05:06,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-25) [78e875d] FINISH, SetVdsStatusVDSCommand, log id: 1d36f782 >2014-10-23 11:05:07,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-25) START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 31c7b8ad >2014-10-23 11:05:07,431 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 11:05:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-25) FINISH, SetHaMaintenanceModeVDSCommand, log id: 31c7b8ad >2014-10-23 11:05:07,525 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-25) Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC11 >2014-10-23 11:05:07,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-25) Correlation ID: 78e875d, Job ID: c59a29cf-5c44-4383-81d0-65d3a864f4f4, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-10-23 11:05:07,535 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-25) Lock freed to object EngineLock [exclusiveLocks= key: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c value: VDS >, sharedLocks= ] >2014-10-23 11:05:07,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 25d7f9ce >2014-10-23 11:05:07,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [576e6757] FINISH, GetHardwareInfoVDSCommand, log id: 25d7f9ce >2014-10-23 11:05:08,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [722eb6ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:05:08,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [722eb6ed] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 1ca059d1 >2014-10-23 11:05:08,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [722eb6ed] FINISH, GetHardwareInfoVDSCommand, log id: 1ca059d1 >2014-10-23 11:05:08,222 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [6ea55765] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: b69b6b27-a15a-4f2b-8a70-3ec0a518efb8 Type: StoragePool >2014-10-23 11:05:08,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6ea55765] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster cluster11 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 11:05:08,658 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) [6ea55765] Using Host monique-vds05.tlv.redhat.com from cluster cluster11 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 11:05:08,809 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-48) [6ea55765] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 11:05:08,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [6ea55765] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, targetVdsId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 6a626a15 >2014-10-23 11:05:09,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [6ea55765] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 6a626a15 >2014-10-23 11:05:09,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [6ea55765] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c), log id: 165796d8 >2014-10-23 11:05:09,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [6ea55765] FINISH, SetMOMPolicyParametersVDSCommand, log id: 165796d8 >2014-10-23 11:05:09,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6ea55765] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 11:05:09,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6ea55765] Correlation ID: 6ea55765, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-10-23 11:05:09,647 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [c97830e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:05:26,236 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [2ff60b93] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:05:26,256 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [2ff60b93] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:05:26,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [2ff60b93] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1461dd8b >2014-10-23 11:05:26,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [2ff60b93] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1461dd8b >2014-10-23 11:05:26,386 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [2ff60b93] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:05:26,570 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:05:26,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1733cf29 >2014-10-23 11:05:26,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] FINISH, ConnectStorageServerVDSCommand, return: {8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d=0}, log id: 1733cf29 >2014-10-23 11:05:26,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[SD1, 242a0cb5-8aa9-4bf6-a6bf-34b2058eba50], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 8de0f78 >2014-10-23 11:05:26,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Failed in CreateStorageDomainVDS method >2014-10-23 11:05:26,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=242a0cb5-8aa9-4bf6-a6bf-34b2058eba50, domainName=SD1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',)]] >2014-10-23 11:05:26,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:05:26,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Command CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[SD1, 242a0cb5-8aa9-4bf6-a6bf-34b2058eba50], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=242a0cb5-8aa9-4bf6-a6bf-34b2058eba50, domainName=SD1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 >2014-10-23 11:05:26,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] FINISH, CreateStorageDomainVDSCommand, log id: 8de0f78 >2014-10-23 11:05:26,690 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=242a0cb5-8aa9-4bf6-a6bf-34b2058eba50, domainName=SD1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:05:26,693 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Command [id=97bfe169-8850-4f9c-888a-8c4d39fc9321]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 242a0cb5-8aa9-4bf6-a6bf-34b2058eba50. >2014-10-23 11:05:26,694 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Command [id=97bfe169-8850-4f9c-888a-8c4d39fc9321]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 242a0cb5-8aa9-4bf6-a6bf-34b2058eba50. >2014-10-23 11:05:26,700 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [dccc0db] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:05:26,706 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [dccc0db] Correlation ID: dccc0db, Job ID: b1f65a30-48d8-4c0c-9438-86bcba2e03f0, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain SD1. (User: admin) >2014-10-23 11:05:27,083 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:05:27,136 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:05:27,160 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Removing connection 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d from database >2014-10-23 11:05:27,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42324cfe >2014-10-23 11:05:27,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Failed in DisconnectStorageServerVDS method >2014-10-23 11:05:27,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d', u'port': u''}]",)]] > >2014-10-23 11:05:27,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:05:27,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d', u'port': u''}]",), code = 452 >2014-10-23 11:05:27,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] FINISH, DisconnectStorageServerVDSCommand, log id: 42324cfe >2014-10-23 11:05:27,279 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:05:27,310 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [58f7bcbe] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: 8a791d78-2fcd-4ce6-8aa2-1faeaf091b8d value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:09:25,820 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [1a3a17a0] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:09:25,841 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [1a3a17a0] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:09:25,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [1a3a17a0] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 316c2f3c >2014-10-23 11:09:26,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [1a3a17a0] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 316c2f3c >2014-10-23 11:09:26,483 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [1a3a17a0] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:09:26,647 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:09:26,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: fb13505f-17b6-4dde-82d0-a2fe58bae4fe, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20ad1c1a >2014-10-23 11:09:26,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] FINISH, ConnectStorageServerVDSCommand, return: {fb13505f-17b6-4dde-82d0-a2fe58bae4fe=0}, log id: 20ad1c1a >2014-10-23 11:09:26,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, f99de591-fa42-4e55-b0e9-f11de5c26d7b], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 71fc19c7 >2014-10-23 11:09:26,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Failed in CreateStorageDomainVDS method >2014-10-23 11:09:26,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=f99de591-fa42-4e55-b0e9-f11de5c26d7b, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',)]] >2014-10-23 11:09:26,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:09:26,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Command CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, f99de591-fa42-4e55-b0e9-f11de5c26d7b], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=f99de591-fa42-4e55-b0e9-f11de5c26d7b, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 >2014-10-23 11:09:26,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] FINISH, CreateStorageDomainVDSCommand, log id: 71fc19c7 >2014-10-23 11:09:26,742 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=f99de591-fa42-4e55-b0e9-f11de5c26d7b, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:09:26,745 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Command [id=fd706f67-e518-4f3a-b68d-a562c096228a]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: f99de591-fa42-4e55-b0e9-f11de5c26d7b. >2014-10-23 11:09:26,747 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Command [id=fd706f67-e518-4f3a-b68d-a562c096228a]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: f99de591-fa42-4e55-b0e9-f11de5c26d7b. >2014-10-23 11:09:26,754 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [35c7bb84] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:09:26,777 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [35c7bb84] Correlation ID: 35c7bb84, Job ID: 8ec5531c-fd19-4865-a518-6ad8867ef1c9, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain sd1. (User: admin) >2014-10-23 11:09:27,118 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [71539430] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: fb13505f-17b6-4dde-82d0-a2fe58bae4fe value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:09:27,161 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [71539430] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:09:27,183 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [71539430] Removing connection fb13505f-17b6-4dde-82d0-a2fe58bae4fe from database >2014-10-23 11:09:27,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: fb13505f-17b6-4dde-82d0-a2fe58bae4fe, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f6ce968 >2014-10-23 11:09:27,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] Failed in DisconnectStorageServerVDS method >2014-10-23 11:09:27,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'fb13505f-17b6-4dde-82d0-a2fe58bae4fe', u'port': u''}]",)]] > >2014-10-23 11:09:27,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:09:27,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: fb13505f-17b6-4dde-82d0-a2fe58bae4fe, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'fb13505f-17b6-4dde-82d0-a2fe58bae4fe', u'port': u''}]",), code = 452 >2014-10-23 11:09:27,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [71539430] FINISH, DisconnectStorageServerVDSCommand, log id: 3f6ce968 >2014-10-23 11:09:27,311 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [71539430] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'fb13505f-17b6-4dde-82d0-a2fe58bae4fe', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:09:27,348 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [71539430] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >key: fb13505f-17b6-4dde-82d0-a2fe58bae4fe value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:03,297 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [475be207] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/RHEV/ogofen/8 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:03,311 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [475be207] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:03,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [475be207] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/RHEV/ogofen/8, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 41a0e803 >2014-10-23 11:10:03,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [475be207] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 41a0e803 >2014-10-23 11:10:03,413 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [475be207] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection lion.qa.lab.tlv.redhat.com:/RHEV/ogofen/8 returned by VDSM was: Problem while trying to mount target >2014-10-23 11:10:03,414 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-2) [475be207] The connection with details lion.qa.lab.tlv.redhat.com:/RHEV/ogofen/8 failed because of error code 477 and error message is: problem while trying to mount target >2014-10-23 11:10:03,414 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [475be207] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: ProblemWhileTryingToMountTarget (Failed with error ProblemWhileTryingToMountTarget and code 477) >2014-10-23 11:10:03,418 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [475be207] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-10-23 11:10:03,422 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [475be207] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/RHEV/ogofen/8 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:26,595 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1aa9c9bb] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/8 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:26,646 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1aa9c9bb] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:26,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [1aa9c9bb] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.35.160.108:/RHEV/ogofen/8, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7d024c7c >2014-10-23 11:10:26,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [1aa9c9bb] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 7d024c7c >2014-10-23 11:10:26,758 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1aa9c9bb] Lock freed to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/8 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:26,935 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:26,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: da426776-87f8-4a8e-9b20-b41136a97d28, connection: 10.35.160.108:/RHEV/ogofen/8, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3948e693 >2014-10-23 11:10:26,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] FINISH, ConnectStorageServerVDSCommand, return: {da426776-87f8-4a8e-9b20-b41136a97d28=0}, log id: 3948e693 >2014-10-23 11:10:26,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, e5993c40-1e2a-4a1e-a878-39b5ba217209], args=10.35.160.108:/RHEV/ogofen/8), log id: 388fa81 >2014-10-23 11:10:27,040 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Failed in CreateStorageDomainVDS method >2014-10-23 11:10:27,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=e5993c40-1e2a-4a1e-a878-39b5ba217209, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/8 domVersion=3',)]] >2014-10-23 11:10:27,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:10:27,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Command CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, e5993c40-1e2a-4a1e-a878-39b5ba217209], args=10.35.160.108:/RHEV/ogofen/8) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=e5993c40-1e2a-4a1e-a878-39b5ba217209, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/8 domVersion=3',), code = 351 >2014-10-23 11:10:27,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [3607a848] FINISH, CreateStorageDomainVDSCommand, log id: 388fa81 >2014-10-23 11:10:27,042 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=e5993c40-1e2a-4a1e-a878-39b5ba217209, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/8 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:10:27,045 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Command [id=45f465b0-4105-4eef-a611-e8ba96bef095]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: e5993c40-1e2a-4a1e-a878-39b5ba217209. >2014-10-23 11:10:27,046 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Command [id=45f465b0-4105-4eef-a611-e8ba96bef095]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: e5993c40-1e2a-4a1e-a878-39b5ba217209. >2014-10-23 11:10:27,049 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [3607a848] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:10:27,087 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [3607a848] Correlation ID: 3607a848, Job ID: 94d395c0-f241-40bf-a781-f6e4b5aba932, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain sd1. (User: admin) >2014-10-23 11:10:27,421 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [623237df] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/8 value: STORAGE_CONNECTION >key: da426776-87f8-4a8e-9b20-b41136a97d28 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:27,452 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [623237df] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:27,458 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [623237df] Removing connection da426776-87f8-4a8e-9b20-b41136a97d28 from database >2014-10-23 11:10:27,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: da426776-87f8-4a8e-9b20-b41136a97d28, connection: 10.35.160.108:/RHEV/ogofen/8, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 77542bb1 >2014-10-23 11:10:27,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] Failed in DisconnectStorageServerVDS method >2014-10-23 11:10:27,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/8', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'da426776-87f8-4a8e-9b20-b41136a97d28', u'port': u''}]",)]] > >2014-10-23 11:10:27,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:10:27,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: da426776-87f8-4a8e-9b20-b41136a97d28, connection: 10.35.160.108:/RHEV/ogofen/8, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/8', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'da426776-87f8-4a8e-9b20-b41136a97d28', u'port': u''}]",), code = 452 >2014-10-23 11:10:27,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [623237df] FINISH, DisconnectStorageServerVDSCommand, log id: 77542bb1 >2014-10-23 11:10:27,578 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [623237df] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/8', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'da426776-87f8-4a8e-9b20-b41136a97d28', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:10:27,601 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [623237df] Lock freed to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/8 value: STORAGE_CONNECTION >key: da426776-87f8-4a8e-9b20-b41136a97d28 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:35,312 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [622dbfb6] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/1 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:35,331 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [622dbfb6] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:35,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [622dbfb6] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.35.160.108:/RHEV/ogofen/1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e7291fb >2014-10-23 11:10:35,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [622dbfb6] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6e7291fb >2014-10-23 11:10:35,422 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [622dbfb6] Lock freed to object EngineLock [exclusiveLocks= key: 10.35.160.108:/RHEV/ogofen/1 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:35,565 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:35,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: e181b8f8-3b21-4137-a810-359fd5b034d0, connection: 10.35.160.108:/RHEV/ogofen/1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7dedd29b >2014-10-23 11:10:35,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] FINISH, ConnectStorageServerVDSCommand, return: {e181b8f8-3b21-4137-a810-359fd5b034d0=0}, log id: 7dedd29b >2014-10-23 11:10:35,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, 57dbb575-b41f-4d84-9389-2157bb496d3d], args=10.35.160.108:/RHEV/ogofen/1), log id: 2036da47 >2014-10-23 11:10:35,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Failed in CreateStorageDomainVDS method >2014-10-23 11:10:35,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=57dbb575-b41f-4d84-9389-2157bb496d3d, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/1 domVersion=3',)]] >2014-10-23 11:10:35,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:10:35,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Command CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, 57dbb575-b41f-4d84-9389-2157bb496d3d], args=10.35.160.108:/RHEV/ogofen/1) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=57dbb575-b41f-4d84-9389-2157bb496d3d, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/1 domVersion=3',), code = 351 >2014-10-23 11:10:35,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] FINISH, CreateStorageDomainVDSCommand, log id: 2036da47 >2014-10-23 11:10:35,681 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=57dbb575-b41f-4d84-9389-2157bb496d3d, domainName=sd1, domClass=1, typeSpecificArg=10.35.160.108:/RHEV/ogofen/1 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:10:35,684 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Command [id=edcdbed3-dfd5-4cf7-b223-1434f5480ed5]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 57dbb575-b41f-4d84-9389-2157bb496d3d. >2014-10-23 11:10:35,685 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Command [id=edcdbed3-dfd5-4cf7-b223-1434f5480ed5]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 57dbb575-b41f-4d84-9389-2157bb496d3d. >2014-10-23 11:10:35,694 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [27d430a4] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:10:35,698 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [27d430a4] Correlation ID: 27d430a4, Job ID: c62e9e4d-8d1c-4456-93e3-4134b9835555, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain sd1. (User: admin) >2014-10-23 11:10:36,035 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Lock Acquired to object EngineLock [exclusiveLocks= key: e181b8f8-3b21-4137-a810-359fd5b034d0 value: STORAGE_CONNECTION >key: 10.35.160.108:/RHEV/ogofen/1 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:10:36,054 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:10:36,060 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Removing connection e181b8f8-3b21-4137-a810-359fd5b034d0 from database >2014-10-23 11:10:36,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: e181b8f8-3b21-4137-a810-359fd5b034d0, connection: 10.35.160.108:/RHEV/ogofen/1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6367976f >2014-10-23 11:10:36,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Failed in DisconnectStorageServerVDS method >2014-10-23 11:10:36,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'e181b8f8-3b21-4137-a810-359fd5b034d0', u'port': u''}]",)]] > >2014-10-23 11:10:36,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:10:36,175 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: e181b8f8-3b21-4137-a810-359fd5b034d0, connection: 10.35.160.108:/RHEV/ogofen/1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'e181b8f8-3b21-4137-a810-359fd5b034d0', u'port': u''}]",), code = 452 >2014-10-23 11:10:36,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [39b42082] FINISH, DisconnectStorageServerVDSCommand, log id: 6367976f >2014-10-23 11:10:36,175 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'10.35.160.108:/RHEV/ogofen/1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'e181b8f8-3b21-4137-a810-359fd5b034d0', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:10:36,184 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [39b42082] Lock freed to object EngineLock [exclusiveLocks= key: e181b8f8-3b21-4137-a810-359fd5b034d0 value: STORAGE_CONNECTION >key: 10.35.160.108:/RHEV/ogofen/1 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:13:10,905 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-80) Failed in GetStatsVDS method, for vds: rose07.qa.lab.tlv.redhat.com; host: rose07.qa.lab.tlv.redhat.com >2014-10-23 11:13:10,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetStatsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]) execution failed. Exception: CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource >2014-10-23 11:13:10,910 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.common.businessentities.IVdsEventListener$$$view5.updateSchedulingStats(Unknown Source) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:699) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:652) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 11:13:10,912 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) ResourceManager::refreshVdsRunTimeInfo: Error: CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource, vds = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com >2014-10-23 11:13:10,912 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:331) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeStoredProcAsBatch(SimpleJdbcCallsHandler.java:52) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeStoredProcAsBatch(SimpleJdbcCallsHandler.java:70) [dal.jar:] > at org.ovirt.engine.core.dao.MassOperationsGenericDaoDbFacade.updateAllInBatch(MassOperationsGenericDaoDbFacade.java:60) [dal.jar:] > at org.ovirt.engine.core.dao.MassOperationsGenericDaoDbFacade.updateAllInBatch(MassOperationsGenericDaoDbFacade.java:95) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.saveDataToDb(VdsUpdateRunTimeInfo.java:188) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:519) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 14 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 16 more > >2014-10-23 11:13:10,914 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Failed to refresh VDS , vds = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com, error = org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down, continuing.: org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.common.businessentities.IVdsEventListener$$$view5.updateSchedulingStats(Unknown Source) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:699) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:652) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 11:13:11,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) ResourceManager::RerunFailedCommand: Error: EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down, vds = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c : rose07.qa.lab.tlv.redhat.com >2014-10-23 11:13:11,002 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down: org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.common.businessentities.IVdsEventListener$$$view5.updateSlaPolicies(Unknown Source) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.afterRefreshTreatment(VdsUpdateRunTimeInfo.java:592) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:257) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 11:13:11,005 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-80) Failed to invoke scheduled method onTimer: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.utils.lock.LockManager$$$view2.releaseLock(Unknown Source) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:291) [vdsbroker.jar:] > ... 6 more > >2014-10-23 11:13:11,008 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-80) failed to reschedule the job: org.quartz.SchedulerException: The Scheduler has been shutdown. > at org.quartz.core.QuartzScheduler.validateState(QuartzScheduler.java:749) [quartz.jar:] > at org.quartz.core.QuartzScheduler.rescheduleJob(QuartzScheduler.java:1060) [quartz.jar:] > at org.quartz.impl.StdScheduler.rescheduleJob(StdScheduler.java:312) [quartz.jar:] > at org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl.rescheduleAJob(SchedulerUtilQuartzImpl.java:331) [scheduler.jar:] > at org.ovirt.engine.core.utils.timer.FixedDelayJobListener.jobWasExecuted(FixedDelayJobListener.java:91) [scheduler.jar:] > at org.quartz.core.QuartzScheduler.notifyJobListenersWasExecuted(QuartzScheduler.java:1936) [quartz.jar:] > at org.quartz.core.JobRunShell.notifyJobListenersComplete(JobRunShell.java:361) [quartz.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:235) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 11:13:39,562 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-10-23 11:13:39,565 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-10-23 11:13:39,566 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-10-23 11:13:39,567 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-10-23 11:13:39,567 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-10-23 11:13:39,567 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-10-23 11:13:39,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-10-23 11:13:39,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-10-23 11:13:39,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-10-23 11:13:39,569 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-10-23 11:13:39,569 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_PORT" is "8702". >2014-10-23 11:13:39,569 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-10-23 11:13:39,569 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_HOST" is "localhost". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-10-23 11:13:39,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PORT" is "5432". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED" is "False". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_USER" is "engine". >2014-10-23 11:13:39,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_GROUP" is "ovirt". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-10-23 11:13:39,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_PORT" is "None". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-10-23 11:13:39,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MAX" is "256m". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MIN" is "256m". >2014-10-23 11:13:39,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-10-23 11:13:39,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-10-23 11:13:39,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-10-23 11:13:39,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-10-23 11:13:39,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-10-23 11:13:39,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-10-23 11:13:39,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-10-23 11:13:39,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_NOT_INSTALLED_URL" is "/ovirt-engine/ReportsNotInstalled.html". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_RIGHTCLICK_URL" is "". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-10-23 11:13:39,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_TIME" is "10". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USER" is "ovirt". >2014-10-23 11:13:39,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-10-23 11:13:39,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-10-23 11:13:39,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-10-23 11:13:39,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-10-23 11:13:39,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-10-23 11:13:39,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-10-23 11:13:59,472 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Start initializing Backend >2014-10-23 11:14:00,534 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 47) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-10-23 11:14:00,862 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Running ovirt-engine 3.5.0-0.17.beta.el6ev >2014-10-23 11:14:00,863 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 47) Start initializing dictionaries >2014-10-23 11:14:00,867 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 47) Finished initializing dictionaries >2014-10-23 11:14:00,867 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Start initializing AuditLogCleanupManager >2014-10-23 11:14:00,868 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-10-23 11:14:00,948 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Finished initializing AuditLogCleanupManager >2014-10-23 11:14:00,949 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Start initializing CommandEntityCleanupManager >2014-10-23 11:14:00,949 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-10-23 11:14:00,971 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Finished initializing CommandEntityCleanupManager >2014-10-23 11:14:00,974 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Start initializing TagsDirector >2014-10-23 11:14:01,003 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Tag root added to tree >2014-10-23 11:14:01,040 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Finished initializing TagsDirector >2014-10-23 11:14:01,041 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 47) Start initializing IsoDomainListSyncronizer >2014-10-23 11:14:01,065 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 47) Finished initializing IsoDomainListSyncronizer >2014-10-23 11:14:01,183 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Completed initializing handlers >2014-10-23 11:14:01,270 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Start initializing ErrorTranslatorImpl >2014-10-23 11:14:01,332 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-10-23 11:14:01,334 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Finished initializing ErrorTranslatorImpl >2014-10-23 11:14:01,334 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Start initializing ErrorTranslatorImpl >2014-10-23 11:14:01,335 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Finished initializing ErrorTranslatorImpl >2014-10-23 11:14:01,335 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Mark incomplete jobs as UNKNOWN >2014-10-23 11:14:01,445 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 47) Start initializing JobRepositoryCleanupManager >2014-10-23 11:14:01,446 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 47) Finished initializing JobRepositoryCleanupManager >2014-10-23 11:14:01,446 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 47) Start initializing AutoRecoveryManager >2014-10-23 11:14:01,453 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 47) Finished initializing AutoRecoveryManager >2014-10-23 11:14:01,475 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 47) Start initializing ExecutionMessageDirector >2014-10-23 11:14:01,478 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 47) Finished initializing ExecutionMessageDirector >2014-10-23 11:14:01,705 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Loading extension 'builtin-authn-internal' >2014-10-23 11:14:01,718 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Extension 'builtin-authn-internal' loaded >2014-10-23 11:14:01,722 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Loading extension 'internal' >2014-10-23 11:14:01,724 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Extension 'internal' loaded >2014-10-23 11:14:01,726 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Initializing extension 'builtin-authn-internal' >2014-10-23 11:14:01,726 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Extension 'builtin-authn-internal' initialized >2014-10-23 11:14:01,760 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Initializing extension 'internal' >2014-10-23 11:14:01,761 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Extension 'internal' initialized >2014-10-23 11:14:01,761 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Start of enabled extensions list >2014-10-23 11:14:01,762 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 11:14:01,762 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 11:14:01,762 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 47) End of enabled extensions list >2014-10-23 11:14:02,152 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 47) Start initializing DbUserCacheManager >2014-10-23 11:14:02,153 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 47) Finished initializing DbUserCacheManager >2014-10-23 11:14:02,221 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 47) Initialization of AsyncTaskManager completed successfully. >2014-10-23 11:14:02,235 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Start initializing ResourceManager >2014-10-23 11:14:02,410 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-10-23 11:14:02,669 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Entered VdsManager constructor >2014-10-23 11:14:02,764 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-10-23 11:14:03,312 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) VDS d8177efa-36f1-40fa-b2d6-01cbe63b3982 was added to the Resource Manager >2014-10-23 11:14:03,312 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Entered VdsManager constructor >2014-10-23 11:14:03,354 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-10-23 11:14:03,385 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) VDS 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c was added to the Resource Manager >2014-10-23 11:14:03,386 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Entered VdsManager constructor >2014-10-23 11:14:03,416 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-23 11:14:03,447 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) VDS e9342ea1-ce56-435b-8284-0e4febfec3c1 was added to the Resource Manager >2014-10-23 11:14:03,502 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Finished initializing ResourceManager >2014-10-23 11:14:03,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 47) Initialization of OvfDataUpdater completed successfully. >2014-10-23 11:14:03,513 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Start scheduling to enable vds load balancer >2014-10-23 11:14:03,513 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Finished scheduling to enable vds load balancer >2014-10-23 11:14:03,513 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Start HA Reservation check >2014-10-23 11:14:03,514 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Finished HA Reservation check >2014-10-23 11:14:03,541 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init VM custom properties utilities >2014-10-23 11:14:03,542 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init device custom properties utilities >2014-10-23 11:14:03,547 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initializing Scheduling manager >2014-10-23 11:14:03,549 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-10-23 11:14:03,597 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-10-23 11:14:03,690 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) External scheduler disabled, discovery skipped >2014-10-23 11:14:03,690 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initialized Scheduling manager >2014-10-23 11:14:03,691 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 47) Initializing DWH Heart Beat >2014-10-23 11:14:03,692 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 47) DWH Heart Beat initialized >2014-10-23 11:14:06,915 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 11:14:07,123 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 11:14:07,126 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:14:08,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = e9342ea1-ce56-435b-8284-0e4febfec3c1, vds=Host[monique-vds05.tlv.redhat.com,e9342ea1-ce56-435b-8284-0e4febfec3c1]), log id: 166c578a >2014-10-23 11:14:08,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 166c578a >2014-10-23 11:14:08,927 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-23 11:14:09,481 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [2fc868cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9342ea1-ce56-435b-8284-0e4febfec3c1 Type: VDS >2014-10-23 11:14:16,059 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-7) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-10-23 11:14:16,060 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-7) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-10-23 11:14:16,179 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-7) Invalid locale found in configuration: >2014-10-23 11:14:16,180 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-7) Invalid locale found in configuration: >2014-10-23 11:14:25,550 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginAdminUserCommand internal: false. >2014-10-23 11:14:25,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: UNASSIGNED not exist in string table >2014-10-23 11:14:25,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-10-23 11:14:25,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-10-23 11:14:25,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-10-23 11:14:25,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-10-23 11:14:25,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-10-23 11:14:25,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-10-23 11:14:25,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-10-23 11:14:25,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-10-23 11:14:26,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 11:14:32,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, vds=Host[monique-vds01.tlv.redhat.com,d8177efa-36f1-40fa-b2d6-01cbe63b3982]), log id: 31eb63f1 >2014-10-23 11:14:32,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 31eb63f1 >2014-10-23 11:14:33,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c, vds=Host[rose07.qa.lab.tlv.redhat.com,3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c]), log id: 356d2798 >2014-10-23 11:14:33,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 356d2798 >2014-10-23 11:14:33,661 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds01.tlv.redhat.com >2014-10-23 11:14:33,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [16b38a56] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: d8177efa-36f1-40fa-b2d6-01cbe63b3982 Type: VDS >2014-10-23 11:14:34,372 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-10-23 11:14:34,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [2fd1f85a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3ba5d489-16b3-4e8c-96a6-ee9ecdbe8d5c Type: VDS >2014-10-23 11:14:38,058 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 11:14:38,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 11:15:09,810 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [6a4d907b] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:15:09,849 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [6a4d907b] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:15:09,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [6a4d907b] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 41aa647f >2014-10-23 11:15:10,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [6a4d907b] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 41aa647f >2014-10-23 11:15:10,552 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [6a4d907b] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:15:10,944 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:15:11,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33f07dde-1798-4c28-9608-b0b26f47c462, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: ae2f8dc >2014-10-23 11:15:11,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] FINISH, ConnectStorageServerVDSCommand, return: {33f07dde-1798-4c28-9608-b0b26f47c462=0}, log id: ae2f8dc >2014-10-23 11:15:11,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, d84e584f-6480-42e8-a6ab-dac1632aaac0], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 4a9333a8 >2014-10-23 11:15:11,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Failed in CreateStorageDomainVDS method >2014-10-23 11:15:11,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=351, mMessage=Error creating a storage domain: (u'storageType=1, sdUUID=d84e584f-6480-42e8-a6ab-dac1632aaac0, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',)]] >2014-10-23 11:15:11,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:15:11,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Command CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storageDomain=StorageDomainStatic[sd1, d84e584f-6480-42e8-a6ab-dac1632aaac0], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=d84e584f-6480-42e8-a6ab-dac1632aaac0, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 >2014-10-23 11:15:11,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] FINISH, CreateStorageDomainVDSCommand, log id: 4a9333a8 >2014-10-23 11:15:11,292 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Error creating a storage domain: (u'storageType=1, sdUUID=d84e584f-6480-42e8-a6ab-dac1632aaac0, domainName=sd1, domClass=1, typeSpecificArg=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 domVersion=3',), code = 351 (Failed with error StorageDomainCreationError and code 351) >2014-10-23 11:15:11,394 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Command [id=22470fa4-bf27-4e95-9a77-e9e73163e84c]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: d84e584f-6480-42e8-a6ab-dac1632aaac0. >2014-10-23 11:15:11,424 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Command [id=22470fa4-bf27-4e95-9a77-e9e73163e84c]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: d84e584f-6480-42e8-a6ab-dac1632aaac0. >2014-10-23 11:15:11,482 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [6ff83e56] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2014-10-23 11:15:11,600 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [6ff83e56] Correlation ID: 6ff83e56, Job ID: c67178fc-a486-4253-a1ab-cf049c5a0166, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain sd1. (User: admin) >2014-10-23 11:15:11,906 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Lock Acquired to object EngineLock [exclusiveLocks= key: 33f07dde-1798-4c28-9608-b0b26f47c462 value: STORAGE_CONNECTION >key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:15:11,944 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:15:11,991 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Removing connection 33f07dde-1798-4c28-9608-b0b26f47c462 from database >2014-10-23 11:15:11,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33f07dde-1798-4c28-9608-b0b26f47c462, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39ffb737 >2014-10-23 11:15:12,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Failed in DisconnectStorageServerVDS method >2014-10-23 11:15:12,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand return value > >ServerConnectionStatusReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=452, mMessage=Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33f07dde-1798-4c28-9608-b0b26f47c462', u'port': u''}]",)]] > >2014-10-23 11:15:12,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] HostName = monique-vds01.tlv.redhat.com >2014-10-23 11:15:12,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Command DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = d8177efa-36f1-40fa-b2d6-01cbe63b3982, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 33f07dde-1798-4c28-9608-b0b26f47c462, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33f07dde-1798-4c28-9608-b0b26f47c462', u'port': u''}]",), code = 452 >2014-10-23 11:15:12,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] FINISH, DisconnectStorageServerVDSCommand, log id: 39ffb737 >2014-10-23 11:15:12,134 ERROR [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Command org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DisconnectStorageServerVDS, error = Error storage server disconnection: (u"domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{u'connection': u'lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'33f07dde-1798-4c28-9608-b0b26f47c462', u'port': u''}]",), code = 452 (Failed with error StorageServerDisconnectionError and code 452) >2014-10-23 11:15:12,203 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [60fc7168] Lock freed to object EngineLock [exclusiveLocks= key: 33f07dde-1798-4c28-9608-b0b26f47c462 value: STORAGE_CONNECTION >key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:40:38,341 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-10-23 11:40:38,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-10-23 11:40:38,403 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-10-23 11:40:38,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-10-23 11:40:38,404 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-10-23 11:40:38,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-10-23 11:40:38,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-10-23 11:40:38,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-10-23 11:40:38,405 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-10-23 11:40:38,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-10-23 11:40:38,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_PORT" is "8702". >2014-10-23 11:40:38,406 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_HOST" is "localhost". >2014-10-23 11:40:38,407 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PORT" is "5432". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED" is "False". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_USER" is "engine". >2014-10-23 11:40:38,408 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_GROUP" is "ovirt". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-10-23 11:40:38,409 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_PORT" is "None". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-10-23 11:40:38,410 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MAX" is "256m". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MIN" is "256m". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-10-23 11:40:38,411 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-10-23 11:40:38,412 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-10-23 11:40:38,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-10-23 11:40:38,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-10-23 11:40:38,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-10-23 11:40:38,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_NOT_INSTALLED_URL" is "/ovirt-engine/ReportsNotInstalled.html". >2014-10-23 11:40:38,413 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-10-23 11:40:38,497 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-10-23 11:40:38,543 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_RIGHTCLICK_URL" is "". >2014-10-23 11:40:38,543 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_TIME" is "10". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-10-23 11:40:38,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USER" is "ovirt". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-10-23 11:40:38,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-10-23 11:41:04,052 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 48) Start initializing Backend >2014-10-23 11:41:05,115 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 48) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-10-23 11:41:05,570 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 48) Running ovirt-engine 3.5.0-0.17.beta.el6ev >2014-10-23 11:41:05,571 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 48) Start initializing dictionaries >2014-10-23 11:41:05,576 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 48) Finished initializing dictionaries >2014-10-23 11:41:05,577 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Start initializing AuditLogCleanupManager >2014-10-23 11:41:05,577 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-10-23 11:41:05,637 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Finished initializing AuditLogCleanupManager >2014-10-23 11:41:05,637 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Start initializing CommandEntityCleanupManager >2014-10-23 11:41:05,637 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-10-23 11:41:05,640 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 48) Finished initializing CommandEntityCleanupManager >2014-10-23 11:41:05,663 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 48) Start initializing TagsDirector >2014-10-23 11:41:05,694 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 48) Tag root added to tree >2014-10-23 11:41:05,735 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 48) Finished initializing TagsDirector >2014-10-23 11:41:05,736 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 48) Start initializing IsoDomainListSyncronizer >2014-10-23 11:41:05,760 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 48) Finished initializing IsoDomainListSyncronizer >2014-10-23 11:41:05,895 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 48) Completed initializing handlers >2014-10-23 11:41:05,985 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 48) Start initializing ErrorTranslatorImpl >2014-10-23 11:41:06,051 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 48) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-10-23 11:41:06,053 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 48) Finished initializing ErrorTranslatorImpl >2014-10-23 11:41:06,053 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 48) Start initializing ErrorTranslatorImpl >2014-10-23 11:41:06,053 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 48) Finished initializing ErrorTranslatorImpl >2014-10-23 11:41:06,054 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 48) Mark incomplete jobs as UNKNOWN >2014-10-23 11:41:06,172 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 48) Start initializing JobRepositoryCleanupManager >2014-10-23 11:41:06,173 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 48) Finished initializing JobRepositoryCleanupManager >2014-10-23 11:41:06,174 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 48) Start initializing AutoRecoveryManager >2014-10-23 11:41:06,176 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 48) Finished initializing AutoRecoveryManager >2014-10-23 11:41:06,179 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 48) Start initializing ExecutionMessageDirector >2014-10-23 11:41:06,196 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 48) Finished initializing ExecutionMessageDirector >2014-10-23 11:41:06,363 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Loading extension 'builtin-authn-internal' >2014-10-23 11:41:06,372 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Extension 'builtin-authn-internal' loaded >2014-10-23 11:41:06,395 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Loading extension 'internal' >2014-10-23 11:41:06,398 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Extension 'internal' loaded >2014-10-23 11:41:06,399 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Initializing extension 'builtin-authn-internal' >2014-10-23 11:41:06,400 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Extension 'builtin-authn-internal' initialized >2014-10-23 11:41:06,401 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Initializing extension 'internal' >2014-10-23 11:41:06,401 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Extension 'internal' initialized >2014-10-23 11:41:06,402 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Start of enabled extensions list >2014-10-23 11:41:06,402 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 11:41:06,402 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 11:41:06,402 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 48) End of enabled extensions list >2014-10-23 11:41:06,554 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 58) Start initializing DbUserCacheManager >2014-10-23 11:41:06,563 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 58) Finished initializing DbUserCacheManager >2014-10-23 11:41:06,593 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 58) Initialization of AsyncTaskManager completed successfully. >2014-10-23 11:41:06,596 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 58) Start initializing ResourceManager >2014-10-23 11:41:06,869 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 58) Finished initializing ResourceManager >2014-10-23 11:41:06,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 58) Initialization of OvfDataUpdater completed successfully. >2014-10-23 11:41:06,882 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Start scheduling to enable vds load balancer >2014-10-23 11:41:06,882 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Finished scheduling to enable vds load balancer >2014-10-23 11:41:06,883 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Start HA Reservation check >2014-10-23 11:41:06,883 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Finished HA Reservation check >2014-10-23 11:41:06,913 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-10-23 11:41:06,923 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 58) Init VM custom properties utilities >2014-10-23 11:41:06,925 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 58) Init device custom properties utilities >2014-10-23 11:41:06,931 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Initializing Scheduling manager >2014-10-23 11:41:07,002 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-10-23 11:41:07,149 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) External scheduler disabled, discovery skipped >2014-10-23 11:41:07,150 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 58) Initialized Scheduling manager >2014-10-23 11:41:07,150 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 58) Initializing DWH Heart Beat >2014-10-23 11:41:07,151 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 58) DWH Heart Beat initialized >2014-10-23 11:41:07,384 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-10-23 11:48:37,653 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 11:48:37,654 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 11:48:51,489 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-8) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-10-23 11:48:51,506 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-8) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-10-23 11:48:51,518 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-8) Invalid locale found in configuration: >2014-10-23 11:48:51,541 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-8) Invalid locale found in configuration: >2014-10-23 11:49:44,692 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [18d486a] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:44,696 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [8392490] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:44,722 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [6234d63e] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:44,742 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [183b1022] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:44,744 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [137bf28e] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:44,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: UNASSIGNED not exist in string table >2014-10-23 11:49:44,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-10-23 11:49:44,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-10-23 11:49:44,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-10-23 11:49:44,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-10-23 11:49:44,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-10-23 11:49:44,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-10-23 11:49:44,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-10-23 11:49:44,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-10-23 11:49:45,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [18d486a] Correlation ID: 18d486a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,076 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [137bf28e] Correlation ID: 137bf28e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,077 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [6234d63e] Correlation ID: 6234d63e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [8392490] Correlation ID: 8392490, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [183b1022] Correlation ID: 183b1022, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,211 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [6108b6ef] Running command: LogoutUserCommand internal: false. >2014-10-23 11:49:45,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [6108b6ef] Correlation ID: 6108b6ef, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 11:49:45,341 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-10-23 11:49:45,341 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-10-23 11:49:50,376 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 11:49:51,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 11:50:45,048 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2014-10-23 11:50:45,099 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 11:50:45,245 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 11:55:01,792 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-8) [737ab858] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 11:55:01,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [737ab858] Correlation ID: 737ab858, Call Stack: null, Custom Event ID: -1, Message: Data Center DC1, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-10-23 11:55:14,012 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-4) [40bb8a9a] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePoolAction group CREATE_CLUSTER with role type ADMIN >2014-10-23 11:55:14,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [40bb8a9a] Correlation ID: 40bb8a9a, Call Stack: null, Custom Event ID: -1, Message: Cluster Cluster1 was added by admin >2014-10-23 11:55:47,646 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-11) [29ca1ea6] Running command: AddVdsCommand internal: false. Entities affected : ID: ffba78f7-1ddd-469a-8b49-9240632ea90a Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-10-23 11:55:48,162 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 11:55:48,213 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] Lock freed to object EngineLock [exclusiveLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 11:55:48,403 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] START, RemoveVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 69345926 >2014-10-23 11:55:48,403 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] FINISH, RemoveVdsVDSCommand, log id: 69345926 >2014-10-23 11:55:48,531 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] START, AddVdsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 60255139 >2014-10-23 11:55:48,531 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] AddVds - entered , starting logic to add VDS 2c075add-8452-44b3-872d-20db3a5f1c2b >2014-10-23 11:55:48,610 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] AddVds - VDS 2c075add-8452-44b3-872d-20db3a5f1c2b was added, will try to add it to the resource manager >2014-10-23 11:55:48,627 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-11) [59e6fa32] Entered VdsManager constructor >2014-10-23 11:55:48,662 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-11) [59e6fa32] Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-10-23 11:55:49,376 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-11) [59e6fa32] VDS 2c075add-8452-44b3-872d-20db3a5f1c2b was added to the Resource Manager >2014-10-23 11:55:49,399 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-11) [59e6fa32] FINISH, AddVdsVDSCommand, log id: 60255139 >2014-10-23 11:55:49,945 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-11) [59e6fa32] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-10-23 11:55:50,022 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [59e6fa32] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.There is no other host in the data center that can be used to test the power management settings. >2014-10-23 11:55:50,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [59e6fa32] Correlation ID: 29ca1ea6, Job ID: 85cf72a4-7d8b-4889-adb3-d5004148fb17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was added by admin. >2014-10-23 11:55:50,326 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 11:55:50,359 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 11:55:50,372 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Before Installation host 2c075add-8452-44b3-872d-20db3a5f1c2b, rose07.qa.lab.tlv.redhat.com >2014-10-23 11:55:50,530 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1a0def66 >2014-10-23 11:55:50,606 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] FINISH, SetVdsStatusVDSCommand, log id: 1a0def66 >2014-10-23 11:55:50,789 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Installation rose07.qa.lab.tlv.redhat.com: Connected to host rose07.qa.lab.tlv.redhat.com with SSH key fingerprint: ed:27:fb:75:a2:a0:82:18:3f:90:38:f5:44:3c:ee:61 >2014-10-23 11:55:51,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Connected to host rose07.qa.lab.tlv.redhat.com with SSH key fingerprint: ed:27:fb:75:a2:a0:82:18:3f:90:38:f5:44:3c:ee:61. >2014-10-23 11:55:51,054 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Installation of rose07.qa.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-10-23 11:55:51,078 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-5) SSH execute root@rose07.qa.lab.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-10-23 11:55:52,241 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Initializing >2014-10-23 11:55:52,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Initializing. >2014-10-23 11:55:52,350 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment setup >2014-10-23 11:55:52,467 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-10-23 11:55:52,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment setup. >2014-10-23 11:55:52,469 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment packages setup >2014-10-23 11:55:52,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment packages setup. >2014-10-23 11:55:56,417 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Programs detection >2014-10-23 11:55:56,467 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Programs detection. >2014-10-23 11:55:56,467 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Environment customization >2014-10-23 11:55:56,664 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Environment customization. >2014-10-23 11:55:56,665 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Kdump supported >2014-10-23 11:55:56,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Kdump supported. >2014-10-23 11:55:56,752 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20141023085516-p7c2kc.log' >2014-10-23 11:55:56,818 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20141023085516-p7c2kc.log'. >2014-10-23 11:55:57,028 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host rose07.qa.lab.tlv.redhat.com reports unique id 4C4C4544-0058-5610-8056-B5C04F4D5731 >2014-10-23 11:55:57,031 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0058-5610-8056-B5C04F4D5731 to Host rose07.qa.lab.tlv.redhat.com >2014-10-23 11:55:57,627 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Setup validation >2014-10-23 11:55:57,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Setup validation. >2014-10-23 11:55:57,713 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Hardware supports virtualization >2014-10-23 11:55:57,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Hardware supports virtualization. >2014-10-23 11:55:57,796 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Transaction setup >2014-10-23 11:55:57,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Transaction setup. >2014-10-23 11:55:57,890 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-10-23 11:55:57,973 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 11:55:57,973 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Package installation >2014-10-23 11:55:58,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Package installation. >2014-10-23 11:56:02,964 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Misc configuration >2014-10-23 11:56:03,024 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 11:56:03,037 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Setting time >2014-10-23 11:56:03,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Setting time. >2014-10-23 11:56:04,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Enrolling certificate >2014-10-23 11:56:04,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Enrolling certificate. >2014-10-23 11:56:05,916 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-7) [2178b0c4] Running command: AddVdsCommand internal: false. Entities affected : ID: ffba78f7-1ddd-469a-8b49-9240632ea90a Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-10-23 11:56:05,977 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 11:56:06,003 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] Lock freed to object EngineLock [exclusiveLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 11:56:06,007 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] START, RemoveVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 299959aa >2014-10-23 11:56:06,007 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] FINISH, RemoveVdsVDSCommand, log id: 299959aa >2014-10-23 11:56:06,009 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] START, AddVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 7fa30151 >2014-10-23 11:56:06,010 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] AddVds - entered , starting logic to add VDS acb7e296-b82a-4376-9193-f26aedc8ef44 >2014-10-23 11:56:06,012 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] AddVds - VDS acb7e296-b82a-4376-9193-f26aedc8ef44 was added, will try to add it to the resource manager >2014-10-23 11:56:06,012 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [3384f21c] Entered VdsManager constructor >2014-10-23 11:56:06,045 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [3384f21c] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-23 11:56:06,046 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [3384f21c] VDS acb7e296-b82a-4376-9193-f26aedc8ef44 was added to the Resource Manager >2014-10-23 11:56:06,047 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3384f21c] FINISH, AddVdsVDSCommand, log id: 7fa30151 >2014-10-23 11:56:06,068 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [3384f21c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 11:56:06,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [3384f21c] Correlation ID: 2178b0c4, Job ID: e998e751-73be-4c5b-8068-588f55787898, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was added by admin. >2014-10-23 11:56:06,126 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] Lock Acquired to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 11:56:06,135 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Transaction commit >2014-10-23 11:56:06,163 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 11:56:06,164 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] Before Installation host acb7e296-b82a-4376-9193-f26aedc8ef44, monique-vds05.tlv.redhat.com >2014-10-23 11:56:06,194 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 22e86864 >2014-10-23 11:56:06,210 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] FINISH, SetVdsStatusVDSCommand, log id: 22e86864 >2014-10-23 11:56:06,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Transaction commit. >2014-10-23 11:56:06,318 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-6) [297d9878] Installation monique-vds05.tlv.redhat.com: Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: a6:21:bf:f9:e3:8f:d4:03:34:a3:89:7d:04:8a:d2:30 >2014-10-23 11:56:06,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-6) [297d9878] Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Connected to host monique-vds05.tlv.redhat.com with SSH key fingerprint: a6:21:bf:f9:e3:8f:d4:03:34:a3:89:7d:04:8a:d2:30. >2014-10-23 11:56:06,434 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-6) [297d9878] Installation of monique-vds05.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-10-23 11:56:06,443 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-6) SSH execute root@monique-vds05.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-10-23 11:56:06,848 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Closing up >2014-10-23 11:56:06,976 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Closing up. >2014-10-23 11:56:07,238 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Restarting kdump >2014-10-23 11:56:07,349 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Initializing >2014-10-23 11:56:07,409 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Initializing. >2014-10-23 11:56:07,436 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment setup >2014-10-23 11:56:07,455 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Environment setup. >2014-10-23 11:56:07,455 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment packages setup >2014-10-23 11:56:07,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Restarting kdump. >2014-10-23 11:56:07,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Environment packages setup. >2014-10-23 11:56:08,022 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stopping libvirtd >2014-10-23 11:56:08,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stopping libvirtd. >2014-10-23 11:56:08,246 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Starting vdsm >2014-10-23 11:56:08,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Starting vdsm. >2014-10-23 11:56:09,050 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-23 11:56:10,181 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Pre-termination >2014-10-23 11:56:10,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Pre-termination. >2014-10-23 11:56:10,263 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023115610-rose07.qa.lab.tlv.redhat.com-2e71d4bf.log' >2014-10-23 11:56:10,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023115610-rose07.qa.lab.tlv.redhat.com-2e71d4bf.log'. >2014-10-23 11:56:10,692 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation rose07.qa.lab.tlv.redhat.com: Stage: Termination >2014-10-23 11:56:10,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 2e71d4bf, Call Stack: null, Custom Event ID: -1, Message: Installing Host rose07.qa.lab.tlv.redhat.com. Stage: Termination. >2014-10-23 11:56:10,961 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 11:56:11,717 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Engine managed to communicate with VDSM agent on host rose07.qa.lab.tlv.redhat.com >2014-10-23 11:56:12,075 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Programs detection >2014-10-23 11:56:12,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Programs detection. >2014-10-23 11:56:12,097 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Environment customization >2014-10-23 11:56:12,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Environment customization. >2014-10-23 11:56:12,231 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Kdump supported >2014-10-23 11:56:12,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Kdump supported. >2014-10-23 11:56:12,289 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20141023115610-2u16ce.log' >2014-10-23 11:56:12,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20141023115610-2u16ce.log'. >2014-10-23 11:56:12,421 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds05.tlv.redhat.com reports unique id 4C4C4544-0032-4310-8051-B7C04F39354A >2014-10-23 11:56:12,553 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0032-4310-8051-B7C04F39354A to Host monique-vds05.tlv.redhat.com >2014-10-23 11:56:12,586 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] The management network rhevm is already configured on host rose07.qa.lab.tlv.redhat.com >2014-10-23 11:56:12,643 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 753ca49f >2014-10-23 11:56:12,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] FINISH, SetVdsStatusVDSCommand, log id: 753ca49f >2014-10-23 11:56:12,697 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] After Installation host rose07.qa.lab.tlv.redhat.com, VDS >2014-10-23 11:56:12,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Correlation ID: 2e71d4bf, Job ID: 85cf72a4-7d8b-4889-adb3-d5004148fb17, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com installed >2014-10-23 11:56:12,752 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Disabling Kdump integration >2014-10-23 11:56:12,779 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Disabling Kdump integration. >2014-10-23 11:56:12,794 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-5) [2e71d4bf] Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 11:56:12,925 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Setup validation >2014-10-23 11:56:12,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Setup validation. >2014-10-23 11:56:12,942 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Hardware supports virtualization >2014-10-23 11:56:12,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Hardware supports virtualization. >2014-10-23 11:56:13,018 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction setup >2014-10-23 11:56:13,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Transaction setup. >2014-10-23 11:56:13,074 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-10-23 11:56:13,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 11:56:13,130 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Package installation >2014-10-23 11:56:13,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Package installation. >2014-10-23 11:56:14,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 7aecd5e1 >2014-10-23 11:56:14,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, GetHardwareInfoVDSCommand, log id: 7aecd5e1 >2014-10-23 11:56:14,534 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [15228014] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 11:56:14,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [15228014] START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 460ebb2b >2014-10-23 11:56:14,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [15228014] FINISH, GetHardwareInfoVDSCommand, log id: 460ebb2b >2014-10-23 11:56:14,974 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [eadf84a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 11:56:15,501 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [eadf84a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 11:56:15,501 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) [eadf84a] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 11:56:15,712 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-59) [eadf84a] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 11:56:15,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [eadf84a] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 67faf289 >2014-10-23 11:56:15,753 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:56:15,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.read(IOUtil.java:197) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.process(SSLEngineNioHelper.java:130) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.pendingOperations(SSLClient.java:74) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 11:56:17,263 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Misc configuration >2014-10-23 11:56:17,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 11:56:18,461 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Enrolling certificate >2014-10-23 11:56:18,491 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Enrolling certificate. >2014-10-23 11:56:19,803 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Transaction commit >2014-10-23 11:56:19,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Transaction commit. >2014-10-23 11:56:20,551 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Closing up >2014-10-23 11:56:20,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Closing up. >2014-10-23 11:56:21,552 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stopping libvirtd >2014-10-23 11:56:21,568 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stopping libvirtd. >2014-10-23 11:56:21,863 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Starting vdsm >2014-10-23 11:56:21,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Starting vdsm. >2014-10-23 11:56:24,870 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Pre-termination >2014-10-23 11:56:24,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Pre-termination. >2014-10-23 11:56:24,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023115624-monique-vds05.tlv.redhat.com-297d9878.log' >2014-10-23 11:56:24,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023115624-monique-vds05.tlv.redhat.com-297d9878.log'. >2014-10-23 11:56:25,144 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds05.tlv.redhat.com: Stage: Termination >2014-10-23 11:56:25,157 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 297d9878, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds05.tlv.redhat.com. Stage: Termination. >2014-10-23 11:56:25,269 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:56:25,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 11:56:25,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 11:56:25,292 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (org.ovirt.thread.pool-7-thread-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:917) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:913) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:405) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:394) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 11:56:25,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-10-23 11:56:25,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] Command PollVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-10-23 11:56:25,824 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:56:26,102 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-6) [297d9878] Engine managed to communicate with VDSM agent on host monique-vds05.tlv.redhat.com >2014-10-23 11:56:26,641 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-6) [297d9878] The management network rhevm is already configured on host monique-vds05.tlv.redhat.com >2014-10-23 11:56:26,647 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 536836e8 >2014-10-23 11:56:26,652 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] FINISH, SetVdsStatusVDSCommand, log id: 536836e8 >2014-10-23 11:56:26,652 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] After Installation host monique-vds05.tlv.redhat.com, VDS >2014-10-23 11:56:26,674 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-6) [297d9878] Correlation ID: 297d9878, Job ID: e998e751-73be-4c5b-8068-588f55787898, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com installed >2014-10-23 11:56:26,680 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-6) [297d9878] Lock freed to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 11:56:27,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vds=Host[monique-vds05.tlv.redhat.com,acb7e296-b82a-4376-9193-f26aedc8ef44]), log id: 42082f6 >2014-10-23 11:56:27,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, GetHardwareInfoVDSCommand, log id: 42082f6 >2014-10-23 11:56:27,318 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [13c27b56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 11:56:27,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [13c27b56] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vds=Host[monique-vds05.tlv.redhat.com,acb7e296-b82a-4376-9193-f26aedc8ef44]), log id: 4a2c574a >2014-10-23 11:56:27,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [13c27b56] FINISH, GetHardwareInfoVDSCommand, log id: 4a2c574a >2014-10-23 11:56:27,503 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-90) [2eb3e6a7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 11:56:27,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-90) [2eb3e6a7] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 485f1716 >2014-10-23 11:56:27,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-90) [2eb3e6a7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 485f1716 >2014-10-23 11:56:27,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [2eb3e6a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-10-23 11:56:27,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [2eb3e6a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 11:56:27,904 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [219fd7da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 11:56:27,938 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [5989052c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 11:56:52,589 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [75a2caca] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:56:52,614 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [75a2caca] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:56:52,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [75a2caca] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 657cf74 >2014-10-23 11:56:53,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [75a2caca] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 657cf74 >2014-10-23 11:56:53,265 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [75a2caca] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 11:56:53,560 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-11) [1486915] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:56:53,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3146309 >2014-10-23 11:56:53,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 3146309 >2014-10-23 11:56:53,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] START, CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storageDomain=StorageDomainStatic[SD1, e22453d1-773c-4f02-8222-a2c0f2aa5f63], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2), log id: 6f63effc >2014-10-23 11:56:54,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] FINISH, CreateStorageDomainVDSCommand, log id: 6f63effc >2014-10-23 11:56:54,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] START, GetStorageDomainStatsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storageDomainId=e22453d1-773c-4f02-8222-a2c0f2aa5f63), log id: 35d7053d >2014-10-23 11:56:54,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-11) [1486915] FINISH, GetStorageDomainStatsVDSCommand, return: StorageDomain[, e22453d1-773c-4f02-8222-a2c0f2aa5f63], log id: 35d7053d >2014-10-23 11:56:54,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [1486915] Correlation ID: 1486915, Job ID: 4d87e298-821f-420d-9fee-506ec09d5846, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD1 was added by admin >2014-10-23 11:56:54,615 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [2195ed9b] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:56:54,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [2195ed9b] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1df9990d >2014-10-23 11:56:54,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [2195ed9b] FINISH, DisconnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 1df9990d >2014-10-23 11:56:55,177 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [1f251dd5] Lock Acquired to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= ] >2014-10-23 11:56:55,362 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [1f251dd5] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:56:55,454 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [44acf142] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 11:56:55,708 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-1) [6de97285] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:56:55,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6de97285] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4d080e85 >2014-10-23 11:56:55,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6de97285] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 4d080e85 >2014-10-23 11:56:55,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [6de97285] START, CreateStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId=f19f3eec-207f-4455-8715-dd11cd054849, storagePoolName=DC1, masterDomainId=e22453d1-773c-4f02-8222-a2c0f2aa5f63, domainsIdList=[e22453d1-773c-4f02-8222-a2c0f2aa5f63], masterVersion=1), log id: 37e9a380 >2014-10-23 11:57:19,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [6de97285] FINISH, CreateStoragePoolVDSCommand, log id: 37e9a380 >2014-10-23 11:57:19,347 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [6de97285] Lock freed to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= ] >2014-10-23 11:57:19,352 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [37ec4eba] Lock Acquired to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= ] >2014-10-23 11:57:19,359 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [37ec4eba] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:57:19,401 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [37ec4eba] Lock freed to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= ] >2014-10-23 11:57:19,402 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [37ec4eba] ActivateStorage Domain. Before Connect all hosts to pool. Time:10/23/14 11:57 AM >2014-10-23 11:57:19,442 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-28) [23012692] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 11:57:19,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [23012692] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ad49662 >2014-10-23 11:57:19,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [23012692] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 3ad49662 >2014-10-23 11:57:19,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [37ec4eba] START, ActivateStorageDomainVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63), log id: 2d7ccd80 >2014-10-23 11:57:19,511 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 11:57:19,532 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Correlation ID: 3a7f5d75, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-10-23 11:57:19,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-1) [3a7f5d75] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC1 >2014-10-23 11:57:19,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vdsId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 4563b917 >2014-10-23 11:57:19,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] FINISH, ConnectStoragePoolVDSCommand, log id: 4563b917 >2014-10-23 11:57:19,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-1) [3a7f5d75] hostFromVds::Connected host to pool - selectedVds - monique-vds05.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-23 11:57:19,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-1) [3a7f5d75] starting spm on vds monique-vds05.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-10-23 11:57:19,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] START, SpmStartVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 465db975 >2014-10-23 11:57:19,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] spmStart polling started: taskId = a4489e46-8aaa-45d2-9bae-af4739800b9a >2014-10-23 11:57:42,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] spmStart polling ended: taskId = a4489e46-8aaa-45d2-9bae-af4739800b9a task status = finished >2014-10-23 11:57:42,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] spmStart polling ended, spm status: SPM >2014-10-23 11:57:42,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=a4489e46-8aaa-45d2-9bae-af4739800b9a), log id: 2ff1b067 >2014-10-23 11:57:42,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] FINISH, HSMClearTaskVDSCommand, log id: 2ff1b067 >2014-10-23 11:57:42,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35f2f655, log id: 465db975 >2014-10-23 11:57:42,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-10-23 11:57:42,360 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05.tlv.redhat.com (Address: monique-vds05.tlv.redhat.com). >2014-10-23 11:57:42,368 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:57:42,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] FINISH, ActivateStorageDomainVDSCommand, return: true, log id: 2d7ccd80 >2014-10-23 11:57:42,492 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] ActivateStorage Domain. After Connect all hosts to pool. Time:10/23/14 11:57 AM >2014-10-23 11:57:42,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-40) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vdsId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 7e48ca87 >2014-10-23 11:57:42,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-40) FINISH, ConnectStoragePoolVDSCommand, log id: 7e48ca87 >2014-10-23 11:57:42,919 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] ActivateStorage Domain. After change storage pool status in vds. Time:10/23/14 11:57 AM >2014-10-23 11:57:43,001 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Correlation ID: 37ec4eba, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD1 (Data Center DC1) was activated by admin >2014-10-23 11:57:43,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Correlation ID: 44acf142, Job ID: 34aa3378-8b75-4435-8e50-de032cf8eb8c, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center DC1 by admin >2014-10-23 11:57:43,076 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Correlation ID: 1f251dd5, Job ID: 34aa3378-8b75-4435-8e50-de032cf8eb8c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD1 was attached to Data Center DC1 by admin >2014-10-23 11:57:43,093 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Trying to release exclusive lock which does not exist, lock key: e22453d1-773c-4f02-8222-a2c0f2aa5f63STORAGE >2014-10-23 11:57:43,093 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [3a7f5d75] Lock freed to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= ] >2014-10-23 11:59:17,996 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 11:59:18,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [eadf84a] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 67faf289 >2014-10-23 11:59:18,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-59) [eadf84a] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 6d650631 >2014-10-23 11:59:18,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-59) [eadf84a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6d650631 >2014-10-23 11:59:18,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [eadf84a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 11:59:18,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [eadf84a] Correlation ID: eadf84a, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-10-23 11:59:18,836 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [54cc1268] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 11:59:18,937 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [238d8ab6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 11:59:53,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-7) domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 in problem. vds: rose07.qa.lab.tlv.redhat.com >2014-10-23 12:04:53,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-48) starting processDomainRecovery for domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 >2014-10-23 12:04:53,464 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-48) vds rose07.qa.lab.tlv.redhat.com reported domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 as in problem, attempting to move the vds to status NonOperational >2014-10-23 12:04:53,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-48) Removing vds [2c075add-8452-44b3-872d-20db3a5f1c2b] from the domain in maintenance cache >2014-10-23 12:04:53,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-48) Removing host(s) [2c075add-8452-44b3-872d-20db3a5f1c2b] from hosts unseen domain report cache >2014-10-23 12:04:53,547 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-7-thread-48) [346259a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 12:04:53,579 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-48) [346259a6] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 73750cc >2014-10-23 12:05:17,214 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-48) [346259a6] FINISH, SetVdsStatusVDSCommand, log id: 73750cc >2014-10-23 12:05:17,218 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-13) domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 in problem. vds: rose07.qa.lab.tlv.redhat.com >2014-10-23 12:05:17,275 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-48) [346259a6] Correlation ID: 346259a6, Job ID: eb15b355-4ea5-4c9b-bf53-a0a3df1b79c8, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot access the Storage Domain(s) SD1 attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-10-23 12:09:07,708 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-11) [7dca1842] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 12:09:07,783 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-19) [7dca1842] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 12:09:07,786 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-19) [7dca1842] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 12:09:07,786 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-19) [7dca1842] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 12:09:07,886 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-19) [7dca1842] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 43c6f70d >2014-10-23 12:09:07,932 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-19) [7dca1842] FINISH, SetVdsStatusVDSCommand, log id: 43c6f70d >2014-10-23 12:09:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-19) START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 5689b634 >2014-10-23 12:09:08,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-19) FINISH, SetHaMaintenanceModeVDSCommand, log id: 5689b634 >2014-10-23 12:09:08,388 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-19) Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 12:09:08,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-19) Correlation ID: 7dca1842, Job ID: 4ff5918a-7fa2-4d18-847b-d099818e9f7c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-10-23 12:09:08,447 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-19) Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 12:09:08,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 76c058d0 >2014-10-23 12:09:08,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GetHardwareInfoVDSCommand, log id: 76c058d0 >2014-10-23 12:09:09,031 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [2c10067e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 12:09:09,055 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [6b1228b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 12:09:09,114 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 12:09:09,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 48eae5a8 >2014-10-23 12:09:09,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 48eae5a8 >2014-10-23 12:09:09,492 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 12:09:09,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 23ada694 >2014-10-23 12:09:10,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 23ada694 >2014-10-23 12:09:35,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4038d00d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:09:35,748 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) [4038d00d] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:09:35,786 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-78) [4038d00d] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:09:35,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 2ae4e00e >2014-10-23 12:09:36,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 2ae4e00e >2014-10-23 12:09:36,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 5c94888e >2014-10-23 12:09:36,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-78) [4038d00d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5c94888e >2014-10-23 12:09:36,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4038d00d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 12:09:36,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4038d00d] Correlation ID: 6b1228b2, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-10-23 12:09:36,562 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [6103af80] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 12:09:36,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-34) Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 recovered from problem. vds: rose07.qa.lab.tlv.redhat.com >2014-10-23 12:09:36,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-34) Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63:SD1 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-10-23 12:14:34,211 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-10-23 12:14:34,284 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-10-23 12:14:34,285 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-10-23 12:14:34,286 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-10-23 12:14:34,286 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-10-23 12:14:34,286 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-10-23 12:14:34,286 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-10-23 12:14:34,287 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-10-23 12:14:34,287 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_PORT" is "8702". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-10-23 12:14:34,288 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_HOST" is "localhost". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PORT" is "5432". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED" is "False". >2014-10-23 12:14:34,289 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_USER" is "engine". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-10-23 12:14:34,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_FQDN" is "slitmano-rhevm.scl.lab.tlv.redhat.com". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_GROUP" is "ovirt". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-10-23 12:14:34,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_PORT" is "None". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MAX" is "256m". >2014-10-23 12:14:34,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MIN" is "256m". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-10-23 12:14:34,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_BASE_URL" is "". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_NOT_INSTALLED_URL" is "/ovirt-engine/ReportsNotInstalled.html". >2014-10-23 12:14:34,294 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface". >2014-10-23 12:14:34,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "". >2014-10-23 12:14:34,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_RIGHTCLICK_URL" is "". >2014-10-23 12:14:34,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true". >2014-10-23 12:14:34,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true". >2014-10-23 12:14:34,295 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-10-23 12:14:34,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_TIME" is "10". >2014-10-23 12:14:34,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-10-23 12:14:34,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-10-23 12:14:34,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-10-23 12:14:34,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USER" is "ovirt". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-10-23 12:14:34,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-10-23 12:14:53,404 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Start initializing Backend >2014-10-23 12:14:54,549 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 43) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-10-23 12:14:54,978 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Running ovirt-engine 3.5.0-0.17.beta.el6ev >2014-10-23 12:14:54,978 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 43) Start initializing dictionaries >2014-10-23 12:14:54,983 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 43) Finished initializing dictionaries >2014-10-23 12:14:54,983 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Start initializing AuditLogCleanupManager >2014-10-23 12:14:54,983 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-10-23 12:14:55,045 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Finished initializing AuditLogCleanupManager >2014-10-23 12:14:55,046 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Start initializing CommandEntityCleanupManager >2014-10-23 12:14:55,046 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-10-23 12:14:55,048 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 43) Finished initializing CommandEntityCleanupManager >2014-10-23 12:14:55,084 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Start initializing TagsDirector >2014-10-23 12:14:55,123 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Tag root added to tree >2014-10-23 12:14:55,153 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 43) Finished initializing TagsDirector >2014-10-23 12:14:55,154 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 43) Start initializing IsoDomainListSyncronizer >2014-10-23 12:14:55,163 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 43) Finished initializing IsoDomainListSyncronizer >2014-10-23 12:14:55,282 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Completed initializing handlers >2014-10-23 12:14:55,386 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Start initializing ErrorTranslatorImpl >2014-10-23 12:14:55,434 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-10-23 12:14:55,435 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Finished initializing ErrorTranslatorImpl >2014-10-23 12:14:55,436 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Start initializing ErrorTranslatorImpl >2014-10-23 12:14:55,437 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 43) Finished initializing ErrorTranslatorImpl >2014-10-23 12:14:55,437 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 43) Mark incomplete jobs as UNKNOWN >2014-10-23 12:14:55,565 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 43) Start initializing JobRepositoryCleanupManager >2014-10-23 12:14:55,566 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 43) Finished initializing JobRepositoryCleanupManager >2014-10-23 12:14:55,566 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 43) Start initializing AutoRecoveryManager >2014-10-23 12:14:55,567 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 43) Finished initializing AutoRecoveryManager >2014-10-23 12:14:55,570 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 43) Start initializing ExecutionMessageDirector >2014-10-23 12:14:55,573 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 43) Finished initializing ExecutionMessageDirector >2014-10-23 12:14:55,763 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Loading extension 'builtin-authn-internal' >2014-10-23 12:14:55,780 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'builtin-authn-internal' loaded >2014-10-23 12:14:55,784 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Loading extension 'internal' >2014-10-23 12:14:55,786 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'internal' loaded >2014-10-23 12:14:55,787 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Initializing extension 'builtin-authn-internal' >2014-10-23 12:14:55,788 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'builtin-authn-internal' initialized >2014-10-23 12:14:55,789 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Initializing extension 'internal' >2014-10-23 12:14:55,789 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Extension 'internal' initialized >2014-10-23 12:14:55,811 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Start of enabled extensions list >2014-10-23 12:14:55,812 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 12:14:55,812 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-10-23 12:14:55,812 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 43) End of enabled extensions list >2014-10-23 12:14:56,159 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-10-23 12:14:56,171 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-6) Invalid locale found in configuration: >2014-10-23 12:14:56,194 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 45) Start initializing DbUserCacheManager >2014-10-23 12:14:56,230 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 45) Finished initializing DbUserCacheManager >2014-10-23 12:14:56,279 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 45) Initialization of AsyncTaskManager completed successfully. >2014-10-23 12:14:56,296 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 45) Start initializing ResourceManager >2014-10-23 12:14:56,866 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 45) Entered VdsManager constructor >2014-10-23 12:14:56,966 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 45) Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-23 12:14:57,784 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 45) VDS acb7e296-b82a-4376-9193-f26aedc8ef44 was added to the Resource Manager >2014-10-23 12:14:57,784 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 45) Entered VdsManager constructor >2014-10-23 12:14:57,824 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 45) Initialize vdsBroker (rose07.qa.lab.tlv.redhat.com,54,321) >2014-10-23 12:14:57,863 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 45) VDS 2c075add-8452-44b3-872d-20db3a5f1c2b was added to the Resource Manager >2014-10-23 12:14:57,971 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 45) Finished initializing ResourceManager >2014-10-23 12:14:57,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 45) Initialization of OvfDataUpdater completed successfully. >2014-10-23 12:14:57,980 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Start scheduling to enable vds load balancer >2014-10-23 12:14:58,008 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Finished scheduling to enable vds load balancer >2014-10-23 12:14:58,008 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Start HA Reservation check >2014-10-23 12:14:58,009 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Finished HA Reservation check >2014-10-23 12:14:58,059 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) Init VM custom properties utilities >2014-10-23 12:14:58,059 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 45) Init device custom properties utilities >2014-10-23 12:14:58,065 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Initializing Scheduling manager >2014-10-23 12:14:58,131 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Start initializing MacPoolManagerRanges >2014-10-23 12:14:58,175 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-7-thread-1) Finished initializing. Available MACs in pool: 256 >2014-10-23 12:14:58,208 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) External scheduler disabled, discovery skipped >2014-10-23 12:14:58,208 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 45) Initialized Scheduling manager >2014-10-23 12:14:58,209 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 45) Initializing DWH Heart Beat >2014-10-23 12:14:58,210 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 45) DWH Heart Beat initialized >2014-10-23 12:14:58,633 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-10-23 12:15:01,426 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 12:15:01,989 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:15:03,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vds=Host[monique-vds05.tlv.redhat.com,acb7e296-b82a-4376-9193-f26aedc8ef44]), log id: 6addf124 >2014-10-23 12:15:03,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GetHardwareInfoVDSCommand, log id: 6addf124 >2014-10-23 12:15:04,035 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-10-23 12:15:04,036 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-10-23 12:15:04,079 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 12:15:04,079 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-23 12:15:04,100 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-23 12:15:04,357 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [704dc0cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 12:15:08,230 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 12:15:08,251 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] Storage Pool f19f3eec-207f-4455-8715-dd11cd054849 - Updating Storage Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63 status from Active to Unknown, reason : null >2014-10-23 12:15:08,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: UNASSIGNED not exist in string table >2014-10-23 12:15:08,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-10-23 12:15:08,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-10-23 12:15:08,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-10-23 12:15:08,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-10-23 12:15:08,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-10-23 12:15:08,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-10-23 12:15:08,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-10-23 12:15:08,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-10-23 12:15:08,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-10-23 12:15:08,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-10-23 12:15:08,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-10-23 12:15:09,813 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] Correlation ID: 2b84d4bf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-10-23 12:15:10,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus SPM, storage pool DC1 >2014-10-23 12:15:10,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-10-23 12:15:10,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [2b84d4bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05.tlv.redhat.com (Address: monique-vds05.tlv.redhat.com). >2014-10-23 12:15:10,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 12:15:11,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false), log id: 69201d6d >2014-10-23 12:15:11,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] -- executeIrsBrokerCommand: Attempting on storage pool f19f3eec-207f-4455-8715-dd11cd054849 >2014-10-23 12:15:11,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] START, HSMGetAllTasksInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 939e6f5 >2014-10-23 12:15:11,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 939e6f5 >2014-10-23 12:15:11,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69201d6d >2014-10-23 12:15:11,286 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-6) [2b84d4bf] Discovered no tasks on Storage Pool DC1 >2014-10-23 12:15:16,599 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginAdminUserCommand internal: false. >2014-10-23 12:15:16,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 12:15:26,158 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 12:15:26,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 12:15:28,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 442c7aaa >2014-10-23 12:15:28,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 442c7aaa >2014-10-23 12:15:29,042 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Initializing Host: rose07.qa.lab.tlv.redhat.com >2014-10-23 12:15:29,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [6e06665c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 12:17:08,717 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) [2cc0768b] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:17:08,892 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-10-23 12:17:08,893 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-10-23 12:17:08,893 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:17:33,807 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-7) [36c28370] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:17:33,850 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-7) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-10-23 12:17:33,850 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-7) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-10-23 12:17:33,850 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-7) Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:18:45,012 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) [60d4952a] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:18:45,103 ERROR [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) Failed to run RestartVdsCommand on vds :rose07.qa.lab.tlv.redhat.com >2014-10-23 12:18:45,142 WARN [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) CanDoAction of action RestartVds failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL,VDS_FENCE_OPERATION_FAILED,VAR__TYPE__HOST,VAR__ACTION__RESTART >2014-10-23 12:18:45,143 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-4) Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:21:24,836 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (ajp-/127.0.0.1:8702-9) [3818ee3b] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:21:25,166 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:21:25,166 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp-/127.0.0.1:8702-9) Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:21:25,266 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] Running command: RestartVdsCommand internal: false. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 12:21:25,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:21:25,682 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:21:25,724 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:21:25,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 42dc3594 >2014-10-23 12:21:26,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [3818ee3b] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 42dc3594 >2014-10-23 12:21:26,498 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Running command: StopVdsCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 12:21:26,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35ae8e8c >2014-10-23 12:21:35,414 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] FINISH, SetVdsStatusVDSCommand, log id: 35ae8e8c >2014-10-23 12:21:35,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:21:35,585 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Stop command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:21:35,618 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Executing <Stop> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:21:35,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Stop, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 2392a03b >2014-10-23 12:21:41,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 2392a03b >2014-10-23 12:21:42,012 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Waiting for vds rose07.qa.lab.tlv.redhat.com to stop >2014-10-23 12:21:47,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:21:47,131 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:21:47,132 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-10-23 12:21:47,166 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:21:47,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 5f925d2f >2014-10-23 12:22:27,288 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.Done >2014-10-23 12:22:27,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, unknown, log id: 5f925d2f >2014-10-23 12:22:27,289 WARN [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Fencing operation failed with proxy host acb7e296-b82a-4376-9193-f26aedc8ef44, trying another proxy... >2014-10-23 12:22:27,577 ERROR [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Failed to run Power Management command on Host rose07.qa.lab.tlv.redhat.com, no running proxy Host was found. >2014-10-23 12:22:27,577 WARN [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Failed to find other proxy to re-run failed fence operation, retrying with the same proxy... >2014-10-23 12:22:27,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:22:27,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:22:27,764 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:22:27,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 79fe9122 >2014-10-23 12:23:07,901 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host rose07.qa.lab.tlv.redhat.com.Done >2014-10-23 12:23:07,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, unknown, log id: 79fe9122 >2014-10-23 12:23:17,902 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Attempt 2 to get vds rose07.qa.lab.tlv.redhat.com status >2014-10-23 12:23:17,955 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:23:17,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 67a3edcb >2014-10-23 12:23:18,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] FINISH, FenceVdsVDSCommand, return: Test Succeeded, off, log id: 67a3edcb >2014-10-23 12:23:18,070 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (org.ovirt.thread.pool-7-thread-46) [5249e17f] vds rose07.qa.lab.tlv.redhat.com status is off >2014-10-23 12:23:18,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [5249e17f] Correlation ID: 5249e17f, Job ID: 4696e996-195d-4244-99ea-f9559ebb2a99, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was stopped by admin. >2014-10-23 12:23:18,231 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-46) [4d95b646] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 12:23:18,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) [4d95b646] Correlation ID: 4d95b646, Job ID: 4696e996-195d-4244-99ea-f9559ebb2a99, Call Stack: null, Custom Event ID: -1, Message: Auto fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-10-23 12:23:18,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:23:18,712 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:23:18,767 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-46) Running command: StartVdsCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 12:23:18,812 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 54f346ac >2014-10-23 12:23:18,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-46) FINISH, SetVdsStatusVDSCommand, log id: 54f346ac >2014-10-23 12:23:19,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:23:19,100 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Start command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:23:19,160 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) Executing <Start> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:23:19,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Start, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 14ee8ed1 >2014-10-23 12:23:19,277 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:21,277 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:21,278 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:21,279 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:21,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:21,292 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:21,432 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:21,432 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:23,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) FINISH, FenceVdsVDSCommand, return: Test Succeeded, initiated, log id: 14ee8ed1 >2014-10-23 12:23:23,516 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-46) Waiting for vds rose07.qa.lab.tlv.redhat.com to start >2014-10-23 12:23:24,519 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:26,519 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:26,519 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:26,520 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:26,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:26,534 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:26,615 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:28,764 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:23:28,765 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:23:28,765 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-46) Attempt 1 to get vds rose07.qa.lab.tlv.redhat.com status >2014-10-23 12:23:28,802 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-7-thread-46) Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:23:28,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 1b867cde >2014-10-23 12:23:28,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-7-thread-46) FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1b867cde >2014-10-23 12:23:28,900 INFO [org.ovirt.engine.core.bll.StartVdsCommand] (org.ovirt.thread.pool-7-thread-46) vds rose07.qa.lab.tlv.redhat.com status is on >2014-10-23 12:23:28,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: 2e26ff91, Job ID: 4696e996-195d-4244-99ea-f9559ebb2a99, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was started by admin. >2014-10-23 12:23:28,935 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: 3818ee3b, Job ID: 4696e996-195d-4244-99ea-f9559ebb2a99, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was restarted by admin. >2014-10-23 12:23:28,960 INFO [org.ovirt.engine.core.bll.RestartVdsCommand] (org.ovirt.thread.pool-7-thread-46) Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 12:23:29,667 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:31,667 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:31,667 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:31,668 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:31,672 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:31,672 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:31,729 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:34,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:36,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:36,832 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:36,832 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:36,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:36,835 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:36,912 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:39,965 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:41,965 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:41,965 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:41,967 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:41,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:41,970 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:42,087 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:45,165 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:47,166 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:47,166 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:47,167 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:47,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:47,171 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:47,230 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:50,331 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:52,331 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:52,331 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:52,332 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:52,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:52,358 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:52,458 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:55,557 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:23:57,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:57,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:57,558 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:23:57,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:23:57,570 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:23:57,636 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:00,726 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:02,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:02,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:02,727 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:02,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:02,730 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:02,778 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:05,832 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:07,833 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:07,833 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:07,834 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:07,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:07,836 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:07,910 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:10,956 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:12,956 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:12,956 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:12,957 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:12,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:12,960 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:13,030 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:16,074 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:18,074 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:18,074 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:18,075 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:18,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:18,085 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:18,130 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:21,178 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:23,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:23,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:23,180 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:23,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:23,191 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:23,257 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:26,304 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:28,305 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:28,305 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:28,306 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:28,324 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:28,324 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:28,458 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:31,562 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:33,562 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:33,562 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:33,563 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:33,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:33,565 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:33,616 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:36,664 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:38,664 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:38,664 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:38,665 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:38,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:38,669 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:38,737 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:41,795 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:41,796 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:41,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:41,797 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:41,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-10-23 12:24:41,800 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:41,911 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:44,963 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:44,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:44,965 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:44,965 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:44,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-10-23 12:24:44,968 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:45,075 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:48,125 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:48,126 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:48,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:48,127 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:48,130 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-10-23 12:24:48,130 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:48,180 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:51,234 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:51,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:51,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:51,236 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:51,239 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-10-23 12:24:51,240 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:51,288 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:54,350 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:24:56,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:56,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:56,352 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:56,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-23 12:24:56,358 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:24:56,517 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:24:59,570 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:25:00,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:25:00,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:25:00,573 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:25:00,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-10-23 12:25:00,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-10-23 12:25:00,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 12:25:03,755 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 12:25:04,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 3203cf9d >2014-10-23 12:25:04,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GetHardwareInfoVDSCommand, log id: 3203cf9d >2014-10-23 12:25:04,525 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [7ade448c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 12:25:04,555 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [2bab74e7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 12:25:04,616 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 12:25:04,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6dcd382e >2014-10-23 12:25:05,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 6dcd382e >2014-10-23 12:25:05,062 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 12:25:05,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-20) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 5a50b3ee >2014-10-23 12:25:05,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 5a50b3ee >2014-10-23 12:25:30,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [359baec4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com from cluster Cluster1 was chosen as a proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 12:25:30,745 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [359baec4] Using Host monique-vds05.tlv.redhat.com from cluster Cluster1 as proxy to execute Status command on Host rose07.qa.lab.tlv.redhat.com >2014-10-23 12:25:30,789 INFO [org.ovirt.engine.core.bll.FenceExecutor] (DefaultQuartzScheduler_Worker-84) [359baec4] Executing <Status> Power Management command, Proxy Host:monique-vds05.tlv.redhat.com, Agent:ipmilan, Target Host:rose07.qa.lab.tlv.redhat.com, Management IP:rose07-mgmt.qa.lab.tlv.redhat.com, User:root, Options:, Fencing policy:null >2014-10-23 12:25:30,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] START, FenceVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, targetVdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, action = Status, ip = rose07-mgmt.qa.lab.tlv.redhat.com, port = , type = ipmilan, user = root, password = ******, options = '', policy = 'null'), log id: 7c836e9b >2014-10-23 12:25:30,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 7c836e9b >2014-10-23 12:25:30,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 17498e96 >2014-10-23 12:25:31,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [359baec4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 17498e96 >2014-10-23 12:25:31,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [359baec4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 12:25:31,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [359baec4] Correlation ID: 2bab74e7, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com power management was verified successfully. >2014-10-23 12:25:31,615 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [2a833418] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 13:01:33,200 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 13:01:33,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:01:33,330 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [4a9f5136] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:01:33,357 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [1d76675a] Running command: LogoutUserCommand internal: false. >2014-10-23 13:01:33,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [1d76675a] Correlation ID: 1d76675a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:01:46,922 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 13:01:46,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:01:50,890 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 13:01:52,284 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [10403da3] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:01:52,292 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [51bf2056] Running command: LogoutUserCommand internal: false. >2014-10-23 13:01:52,336 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [51bf2056] Correlation ID: 51bf2056, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:02:13,245 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 13:02:13,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:02:17,840 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [62c4d5af] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:02:17,860 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [4fabe195] Running command: LogoutUserCommand internal: false. >2014-10-23 13:02:17,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [4fabe195] Correlation ID: 4fabe195, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:02:46,956 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 13:02:46,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:02:49,998 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 13:02:50,079 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [33eafd11] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:02:50,083 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [6b3953] Running command: LogoutUserCommand internal: false. >2014-10-23 13:02:50,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [6b3953] Correlation ID: 6b3953, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:03:11,640 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 13:03:11,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:03:11,677 INFO [org.ovirt.engine.core.bll.AddExternalJobCommand] (ajp-/127.0.0.1:8702-9) [44d5f5a7] Running command: AddExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 13:03:11,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [44d5f5a7] Correlation ID: 44d5f5a7, Call Stack: null, Custom Event ID: -1, Message: New external Job parent POM for all oVirt modules was added by user admin >2014-10-23 13:03:13,543 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [39065cb2] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:03:13,558 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [22f1b86b] Running command: LogoutUserCommand internal: false. >2014-10-23 13:03:13,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [22f1b86b] Correlation ID: 22f1b86b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:03:40,537 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 13:03:40,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:03:40,585 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [7eddee91] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:03:40,588 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [3a62a1c6] Running command: LogoutUserCommand internal: false. >2014-10-23 13:03:40,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [3a62a1c6] Correlation ID: 3a62a1c6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:03:46,975 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 13:03:46,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:05:13,476 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 13:05:13,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:05:13,581 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [7bd91870] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:05:13,585 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [abe4c56] Running command: LogoutUserCommand internal: false. >2014-10-23 13:05:13,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [abe4c56] Correlation ID: abe4c56, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:05:47,010 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 13:05:47,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:05:50,096 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 13:05:50,121 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [45079f0b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:05:50,137 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [67dad3cb] Running command: LogoutUserCommand internal: false. >2014-10-23 13:05:50,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [67dad3cb] Correlation ID: 67dad3cb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:05:56,501 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 13:05:56,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:05:56,586 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [69100c8a] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:05:56,590 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [c2270e6] Running command: LogoutUserCommand internal: false. >2014-10-23 13:05:56,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [c2270e6] Correlation ID: c2270e6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:06:06,613 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 13:06:06,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:06:07,725 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [4a35456] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:06:07,739 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [3018bc8f] Running command: LogoutUserCommand internal: false. >2014-10-23 13:06:07,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [3018bc8f] Correlation ID: 3018bc8f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:06:07,909 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 13:06:07,927 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [28b33388] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:06:07,974 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [715b1595] Running command: LogoutUserCommand internal: false. >2014-10-23 13:06:07,999 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [715b1595] Correlation ID: 715b1595, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:06:47,006 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 13:06:47,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:07:10,540 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 13:07:10,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:07:10,598 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [69db3097] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:07:10,601 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [67d2a060] Running command: LogoutUserCommand internal: false. >2014-10-23 13:07:10,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [67d2a060] Correlation ID: 67d2a060, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:07:47,033 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 13:07:47,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:08:29,319 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 13:08:29,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:08:29,359 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [29cf0aef] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:08:29,363 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [716450e9] Running command: LogoutUserCommand internal: false. >2014-10-23 13:08:29,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [716450e9] Correlation ID: 716450e9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:08:47,080 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 13:08:47,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:09:17,624 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 13:09:17,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:09:17,666 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [1d147373] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:09:17,670 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [55e86bb1] Running command: LogoutUserCommand internal: false. >2014-10-23 13:09:17,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [55e86bb1] Correlation ID: 55e86bb1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:09:47,089 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 13:09:47,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:09:51,547 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 13:09:51,581 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [c0bd5c3] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:09:51,615 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [1fde7436] Running command: LogoutUserCommand internal: false. >2014-10-23 13:09:51,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [1fde7436] Correlation ID: 1fde7436, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:10:47,090 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 13:10:47,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:12:58,710 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 13:12:58,736 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:12:58,770 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [3970335c] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:12:58,783 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [550c360b] Running command: LogoutUserCommand internal: false. >2014-10-23 13:12:58,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [550c360b] Correlation ID: 550c360b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:13:47,151 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 13:13:47,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:14:57,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-10-23 13:14:58,061 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [b9ad4b2] Lock Acquired to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: OVF_UPDATE >] >2014-10-23 13:14:58,074 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [b9ad4b2] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:14:58,085 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [6979f7d1] Running command: CreateOvfStoresForStorageDomainCommand internal: true. >2014-10-23 13:14:58,096 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [2e6248af] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:14:58,433 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-42) [6e60645e] Running command: AddDiskCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 13:14:58,702 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-10-23 13:14:59,006 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-42) [22eb9e53] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:14:59,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [22eb9e53] START, CreateImageVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 50155417-f62e-471e-a42e-8f22b0438d2d, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1, newImageDescription = OVF_STORE), log id: 7949718 >2014-10-23 13:14:59,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [22eb9e53] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 13:14:59,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [22eb9e53] FINISH, CreateImageVDSCommand, return: 19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1, log id: 7949718 >2014-10-23 13:14:59,475 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-42) [22eb9e53] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 9050aecd-ca7a-4804-9afe-a25da1572743 >2014-10-23 13:14:59,475 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-42) [22eb9e53] CommandMultiAsyncTasks::AttachTask: Attaching task 9e780cd7-6035-4ca3-8484-3adeba10ee5a to command 9050aecd-ca7a-4804-9afe-a25da1572743. >2014-10-23 13:14:59,573 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-42) [22eb9e53] Adding task 9e780cd7-6035-4ca3-8484-3adeba10ee5a (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 13:14:59,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [22eb9e53] Correlation ID: 6e60645e, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 13:14:59,710 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [3f4098b8] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:14:59,816 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-42) [1c8b185] Running command: AddDiskCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 13:14:59,965 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-42) [689753f4] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:15:00,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [689753f4] START, CreateImageVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 2c320404-a668-48c0-9604-e7eacc34d93b, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 42dc2ad2-9fad-4314-a73a-62bb9fb658ed, newImageDescription = OVF_STORE), log id: 36598b63 >2014-10-23 13:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [689753f4] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 13:15:00,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-42) [689753f4] FINISH, CreateImageVDSCommand, return: 42dc2ad2-9fad-4314-a73a-62bb9fb658ed, log id: 36598b63 >2014-10-23 13:15:00,535 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-42) [689753f4] CommandMultiAsyncTasks::AttachTask: Attaching task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd to command 9050aecd-ca7a-4804-9afe-a25da1572743. >2014-10-23 13:15:00,564 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-42) [689753f4] Adding task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 13:15:00,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [689753f4] Correlation ID: 1c8b185, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 13:15:00,615 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] BaseAsyncTask::startPollingTask: Starting to poll task 9e780cd7-6035-4ca3-8484-3adeba10ee5a. >2014-10-23 13:15:00,616 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] BaseAsyncTask::startPollingTask: Starting to poll task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd. >2014-10-23 13:15:00,616 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-42) [689753f4] Lock freed to object EngineLock [exclusiveLocks= key: e22453d1-773c-4f02-8222-a2c0f2aa5f63 value: STORAGE >, sharedLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: OVF_UPDATE >] >2014-10-23 13:15:06,743 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-8) [6e06665c] Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-10-23 13:15:06,849 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-8) [6e06665c] SPMAsyncTask::PollTask: Polling task 9e780cd7-6035-4ca3-8484-3adeba10ee5a (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-10-23 13:15:06,850 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-8) [6e06665c] SPMAsyncTask::PollTask: Polling task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-10-23 13:15:06,850 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-8) [6e06665c] Finished polling Tasks, will poll again in 10 seconds. >2014-10-23 13:15:40,101 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] SPMAsyncTask::PollTask: Polling task 9e780cd7-6035-4ca3-8484-3adeba10ee5a (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 13:15:40,101 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] BaseAsyncTask::onTaskEndSuccess: Task 9e780cd7-6035-4ca3-8484-3adeba10ee5a (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 13:15:40,102 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-42) [689753f4] Task with DB Task ID 77ea7aa9-66ce-4f5a-90a9-61accc8ffc1e and VDSM Task ID 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd is in state Polling. End action for command 9050aecd-ca7a-4804-9afe-a25da1572743 will proceed when all the entitys tasks are completed. >2014-10-23 13:15:40,102 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] SPMAsyncTask::PollTask: Polling task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 13:15:40,102 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] BaseAsyncTask::onTaskEndSuccess: Task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 13:15:40,102 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] CommandAsyncTask::endActionIfNecessary: All tasks of command 9050aecd-ca7a-4804-9afe-a25da1572743 has ended -> executing endAction >2014-10-23 13:15:40,103 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-42) [689753f4] CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 9050aecd-ca7a-4804-9afe-a25da1572743): calling endAction . >2014-10-23 13:15:40,114 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-10) [689753f4] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction CreateOvfStoresForStorageDomain, executionIndex: 0 >2014-10-23 13:15:40,122 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] Ending command successfully: org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand >2014-10-23 13:15:40,252 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-10) [689753f4] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 13:15:40,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [689753f4] START, GetImageInfoVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 2c320404-a668-48c0-9604-e7eacc34d93b, imageId = 42dc2ad2-9fad-4314-a73a-62bb9fb658ed), log id: 64faaa1e >2014-10-23 13:15:41,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [689753f4] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@603841f5, log id: 64faaa1e >2014-10-23 13:15:41,133 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-10) [22eb9e53] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 13:15:41,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [22eb9e53] START, GetImageInfoVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 50155417-f62e-471e-a42e-8f22b0438d2d, imageId = 19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1), log id: 459ba454 >2014-10-23 13:15:41,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-10) [22eb9e53] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@6ba430c6, log id: 459ba454 >2014-10-23 13:15:41,352 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:15:41,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] START, SetVolumeDescriptionVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 2c320404-a668-48c0-9604-e7eacc34d93b, imageId = 42dc2ad2-9fad-4314-a73a-62bb9fb658ed), log id: 31fbf8ed >2014-10-23 13:15:41,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 13:15:41,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] -- setVolumeDescription parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGroupGUID=2c320404-a668-48c0-9604-e7eacc34d93b > volUUID=42dc2ad2-9fad-4314-a73a-62bb9fb658ed > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"e22453d1-773c-4f02-8222-a2c0f2aa5f63"}],"Last Updated":null} > >2014-10-23 13:15:41,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [6979f7d1] FINISH, SetVolumeDescriptionVDSCommand, log id: 31fbf8ed >2014-10-23 13:15:41,980 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 13:15:41,998 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] Running command: UploadStreamCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:15:42,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] START, UploadStreamVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 3cd138c3 >2014-10-23 13:15:42,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- executeVdsBrokerCommand: >2014-10-23 13:15:42,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGUID=2c320404-a668-48c0-9604-e7eacc34d93b > volUUID=42dc2ad2-9fad-4314-a73a-62bb9fb658ed > size=10,240 > >2014-10-23 13:15:45,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] FINISH, UploadStreamVDSCommand, log id: 3cd138c3 >2014-10-23 13:15:45,736 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-10) [5da87942] CommandMultiAsyncTasks::AttachTask: Attaching task b6d4e554-da4d-480d-955f-aaabc4a98cd1 to command 9050aecd-ca7a-4804-9afe-a25da1572743. >2014-10-23 13:15:45,763 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-10) [5da87942] Adding task b6d4e554-da4d-480d-955f-aaabc4a98cd1 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 13:15:45,776 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 13:15:45,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] START, SetVolumeDescriptionVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 2c320404-a668-48c0-9604-e7eacc34d93b, imageId = 42dc2ad2-9fad-4314-a73a-62bb9fb658ed), log id: 7e8a548c >2014-10-23 13:15:45,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 13:15:45,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- setVolumeDescription parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGroupGUID=2c320404-a668-48c0-9604-e7eacc34d93b > volUUID=42dc2ad2-9fad-4314-a73a-62bb9fb658ed > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"e22453d1-773c-4f02-8222-a2c0f2aa5f63"}],"Last Updated":"Thu Oct 23 13:15:41 IDT 2014","Size":10240} > >2014-10-23 13:15:45,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] FINISH, SetVolumeDescriptionVDSCommand, log id: 7e8a548c >2014-10-23 13:15:45,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] START, SetVolumeDescriptionVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 50155417-f62e-471e-a42e-8f22b0438d2d, imageId = 19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1), log id: 16576a9c >2014-10-23 13:15:45,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 13:15:45,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] -- setVolumeDescription parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGroupGUID=50155417-f62e-471e-a42e-8f22b0438d2d > volUUID=19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"e22453d1-773c-4f02-8222-a2c0f2aa5f63"}],"Last Updated":null} > >2014-10-23 13:15:46,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [5da87942] FINISH, SetVolumeDescriptionVDSCommand, log id: 16576a9c >2014-10-23 13:15:46,427 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 13:15:46,448 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] Running command: UploadStreamCommand internal: true. Entities affected : ID: e22453d1-773c-4f02-8222-a2c0f2aa5f63 Type: Storage >2014-10-23 13:15:46,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, UploadStreamVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 6407b24 >2014-10-23 13:15:46,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] -- executeVdsBrokerCommand: >2014-10-23 13:15:46,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] -- parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGUID=50155417-f62e-471e-a42e-8f22b0438d2d > volUUID=19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1 > size=10,240 > >2014-10-23 13:15:47,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, UploadStreamVDSCommand, log id: 6407b24 >2014-10-23 13:15:47,982 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-10) [4289e859] CommandMultiAsyncTasks::AttachTask: Attaching task 0b26a12d-1476-451f-8f90-cabfce59f2ff to command 9050aecd-ca7a-4804-9afe-a25da1572743. >2014-10-23 13:15:48,012 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-10) [4289e859] Adding task 0b26a12d-1476-451f-8f90-cabfce59f2ff (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 13:15:48,016 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 13:15:48,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, SetVolumeDescriptionVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, storageDomainId = e22453d1-773c-4f02-8222-a2c0f2aa5f63, imageGroupId = 50155417-f62e-471e-a42e-8f22b0438d2d, imageId = 19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1), log id: 2ea05ddd >2014-10-23 13:15:48,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 13:15:48,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] -- setVolumeDescription parameters: > spUUID=f19f3eec-207f-4455-8715-dd11cd054849 > sdUUID=e22453d1-773c-4f02-8222-a2c0f2aa5f63 > imageGroupGUID=50155417-f62e-471e-a42e-8f22b0438d2d > volUUID=19d4ea59-0dd7-4b7a-a7ca-1b6cac8ce7a1 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"e22453d1-773c-4f02-8222-a2c0f2aa5f63"}],"Last Updated":"Thu Oct 23 13:15:41 IDT 2014","Size":10240} > >2014-10-23 13:15:48,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, SetVolumeDescriptionVDSCommand, log id: 2ea05ddd >2014-10-23 13:15:48,237 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] BaseAsyncTask::startPollingTask: Starting to poll task b6d4e554-da4d-480d-955f-aaabc4a98cd1. >2014-10-23 13:15:48,237 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] BaseAsyncTask::startPollingTask: Starting to poll task 0b26a12d-1476-451f-8f90-cabfce59f2ff. >2014-10-23 13:15:48,237 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain completed, handling the result. >2014-10-23 13:15:48,237 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain succeeded, clearing tasks. >2014-10-23 13:15:48,237 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd >2014-10-23 13:15:48,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, SPMClearTaskVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, taskId = 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd), log id: 36633c19 >2014-10-23 13:15:48,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=1a85dd34-e93b-4d0d-9442-ebe9c43f44fd), log id: 3e9c1dbe >2014-10-23 13:15:49,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, HSMClearTaskVDSCommand, log id: 3e9c1dbe >2014-10-23 13:15:49,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, SPMClearTaskVDSCommand, log id: 36633c19 >2014-10-23 13:15:49,092 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] BaseAsyncTask::removeTaskFromDB: Removed task 1a85dd34-e93b-4d0d-9442-ebe9c43f44fd from DataBase >2014-10-23 13:15:49,092 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 9e780cd7-6035-4ca3-8484-3adeba10ee5a >2014-10-23 13:15:49,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, SPMClearTaskVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, taskId = 9e780cd7-6035-4ca3-8484-3adeba10ee5a), log id: 4274d10b >2014-10-23 13:15:49,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=9e780cd7-6035-4ca3-8484-3adeba10ee5a), log id: 3484b43 >2014-10-23 13:15:49,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, HSMClearTaskVDSCommand, log id: 3484b43 >2014-10-23 13:15:49,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-10) [4289e859] FINISH, SPMClearTaskVDSCommand, log id: 4274d10b >2014-10-23 13:15:49,354 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-10) [4289e859] BaseAsyncTask::removeTaskFromDB: Removed task 9e780cd7-6035-4ca3-8484-3adeba10ee5a from DataBase >2014-10-23 13:15:49,354 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-10) [4289e859] [within thread]: Some of the tasks related to command id 9050aecd-ca7a-4804-9afe-a25da1572743 were not cleared yet (Task id b6d4e554-da4d-480d-955f-aaabc4a98cd1 is in state Polling). >2014-10-23 13:15:50,110 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-12) Polling and updating Async Tasks: 4 tasks, 2 tasks to poll now >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-12) SPMAsyncTask::PollTask: Polling task 0b26a12d-1476-451f-8f90-cabfce59f2ff (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-12) BaseAsyncTask::onTaskEndSuccess: Task 0b26a12d-1476-451f-8f90-cabfce59f2ff (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-12) Task with DB Task ID 272cdbc1-d2a0-4b94-876f-e48e4621c439 and VDSM Task ID b6d4e554-da4d-480d-955f-aaabc4a98cd1 is in state Polling. End action for command 9050aecd-ca7a-4804-9afe-a25da1572743 will proceed when all the entitys tasks are completed. >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-12) SPMAsyncTask::PollTask: Polling task b6d4e554-da4d-480d-955f-aaabc4a98cd1 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-12) BaseAsyncTask::onTaskEndSuccess: Task b6d4e554-da4d-480d-955f-aaabc4a98cd1 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 13:15:50,422 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-12) CommandAsyncTask::endActionIfNecessary: All tasks of command 9050aecd-ca7a-4804-9afe-a25da1572743 has ended -> executing endAction >2014-10-23 13:15:50,423 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-12) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 9050aecd-ca7a-4804-9afe-a25da1572743): calling endAction . >2014-10-23 13:15:50,423 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-22) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-10-23 13:15:50,623 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-10-23 13:15:50,624 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-10-23 13:15:50,624 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-10-23 13:15:50,624 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] SPMAsyncTask::ClearAsyncTask: Attempting to clear task b6d4e554-da4d-480d-955f-aaabc4a98cd1 >2014-10-23 13:15:50,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] START, SPMClearTaskVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, taskId = b6d4e554-da4d-480d-955f-aaabc4a98cd1), log id: a6e41cd >2014-10-23 13:15:50,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=b6d4e554-da4d-480d-955f-aaabc4a98cd1), log id: 1f71a15a >2014-10-23 13:15:50,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] FINISH, HSMClearTaskVDSCommand, log id: 1f71a15a >2014-10-23 13:15:50,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] FINISH, SPMClearTaskVDSCommand, log id: a6e41cd >2014-10-23 13:15:51,001 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] BaseAsyncTask::removeTaskFromDB: Removed task b6d4e554-da4d-480d-955f-aaabc4a98cd1 from DataBase >2014-10-23 13:15:51,001 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 0b26a12d-1476-451f-8f90-cabfce59f2ff >2014-10-23 13:15:51,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] START, SPMClearTaskVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, taskId = 0b26a12d-1476-451f-8f90-cabfce59f2ff), log id: 5f8229e0 >2014-10-23 13:15:51,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=0b26a12d-1476-451f-8f90-cabfce59f2ff), log id: 5a8b621c >2014-10-23 13:15:51,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] FINISH, HSMClearTaskVDSCommand, log id: 5a8b621c >2014-10-23 13:15:51,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] FINISH, SPMClearTaskVDSCommand, log id: 5f8229e0 >2014-10-23 13:15:51,120 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] BaseAsyncTask::removeTaskFromDB: Removed task 0b26a12d-1476-451f-8f90-cabfce59f2ff from DataBase >2014-10-23 13:15:51,120 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-22) [6979f7d1] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 9050aecd-ca7a-4804-9afe-a25da1572743 >2014-10-23 13:16:21,038 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 13:16:21,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:16:21,081 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [3b41e731] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:16:21,092 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [3a96e8e2] Running command: LogoutUserCommand internal: false. >2014-10-23 13:16:21,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [3a96e8e2] Correlation ID: 3a96e8e2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:16:47,209 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 13:16:47,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:16:53,190 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 13:16:53,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 13:16:53,231 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [7c4062b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 13:16:53,235 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [14723b1d] Running command: LogoutUserCommand internal: false. >2014-10-23 13:16:53,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [14723b1d] Correlation ID: 14723b1d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 13:16:56,367 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Setting new tasks map. The map contains now 0 tasks >2014-10-23 13:16:56,368 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Cleared all tasks of pool f19f3eec-207f-4455-8715-dd11cd054849. >2014-10-23 13:17:47,302 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 13:17:47,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 14:14:58,734 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-10-23 14:15:00,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-10-23 14:40:56,410 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-6) [401d5dbb] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 14:40:56,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [401d5dbb] Correlation ID: 401d5dbb, Call Stack: null, Custom Event ID: -1, Message: Data Center DC2, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-10-23 14:41:08,940 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-1) [6645f8b4] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 6705343f-9285-4942-be95-21ea33bb0125 Type: StoragePoolAction group CREATE_CLUSTER with role type ADMIN >2014-10-23 14:41:09,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [6645f8b4] Correlation ID: 6645f8b4, Call Stack: null, Custom Event ID: -1, Message: Cluster Cluster2 was added by admin >2014-10-23 14:41:19,377 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-2) [3ca8f494] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 14:41:19,394 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3ca8f494] Correlation ID: 3ca8f494, Call Stack: null, Custom Event ID: -1, Message: Data Center DC3, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-10-23 14:41:27,663 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-3) [222f92e9] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: d4869232-1f76-44ed-9c03-2755258c43d5 Type: StoragePoolAction group CREATE_CLUSTER with role type ADMIN >2014-10-23 14:41:27,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [222f92e9] Correlation ID: 222f92e9, Call Stack: null, Custom Event ID: -1, Message: Cluster Cluster3 was added by admin >2014-10-23 14:41:45,461 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-7) [156d3df] Running command: AddVdsCommand internal: false. Entities affected : ID: ee41f71c-7a42-4d1c-9007-11093a1bffb5 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-10-23 14:41:45,502 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 38db43ff-c477-42e3-8fa2-780b6bf164cd Type: VDS >2014-10-23 14:41:45,553 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] Lock freed to object EngineLock [exclusiveLocks= key: d4869232-1f76-44ed-9c03-2755258c43d5 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 14:41:45,561 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] START, RemoveVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd), log id: 15ac01e >2014-10-23 14:41:45,562 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] FINISH, RemoveVdsVDSCommand, log id: 15ac01e >2014-10-23 14:41:45,564 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] START, AddVdsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd), log id: 534ac54d >2014-10-23 14:41:45,565 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] AddVds - entered , starting logic to add VDS 38db43ff-c477-42e3-8fa2-780b6bf164cd >2014-10-23 14:41:45,566 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] AddVds - VDS 38db43ff-c477-42e3-8fa2-780b6bf164cd was added, will try to add it to the resource manager >2014-10-23 14:41:45,566 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [643b84c0] Entered VdsManager constructor >2014-10-23 14:41:45,592 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [643b84c0] Initialize vdsBroker (monique-vds01.tlv.redhat.com,54,321) >2014-10-23 14:41:45,593 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [643b84c0] VDS 38db43ff-c477-42e3-8fa2-780b6bf164cd was added to the Resource Manager >2014-10-23 14:41:45,593 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [643b84c0] FINISH, AddVdsVDSCommand, log id: 534ac54d >2014-10-23 14:41:45,605 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [643b84c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-10-23 14:41:45,654 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Lock Acquired to object EngineLock [exclusiveLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS >, sharedLocks= ] >2014-10-23 14:41:45,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [643b84c0] Correlation ID: 156d3df, Job ID: 017858cd-a613-4228-94fa-df04f422b109, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com was added by admin. >2014-10-23 14:41:45,709 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 38db43ff-c477-42e3-8fa2-780b6bf164cd Type: VDS >2014-10-23 14:41:45,717 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Before Installation host 38db43ff-c477-42e3-8fa2-780b6bf164cd, monique-vds01.tlv.redhat.com >2014-10-23 14:41:45,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d918abe >2014-10-23 14:41:45,760 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] FINISH, SetVdsStatusVDSCommand, log id: 1d918abe >2014-10-23 14:41:45,834 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Installation monique-vds01.tlv.redhat.com: Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03 >2014-10-23 14:41:45,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Connected to host monique-vds01.tlv.redhat.com with SSH key fingerprint: c3:00:a1:d0:75:4d:b0:28:cc:6e:fc:c7:e3:7c:08:03. >2014-10-23 14:41:45,927 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Installation of monique-vds01.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-10-23 14:41:45,978 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-7-thread-49) SSH execute root@monique-vds01.tlv.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-10-23 14:41:47,417 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Initializing >2014-10-23 14:41:47,466 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Initializing. >2014-10-23 14:41:47,489 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment setup >2014-10-23 14:41:47,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment setup. >2014-10-23 14:41:47,500 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment packages setup >2014-10-23 14:41:47,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment packages setup. >2014-10-23 14:41:48,597 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Initializing Host: monique-vds01.tlv.redhat.com >2014-10-23 14:41:59,021 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Programs detection >2014-10-23 14:41:59,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Programs detection. >2014-10-23 14:41:59,031 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Environment customization >2014-10-23 14:41:59,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Environment customization. >2014-10-23 14:41:59,357 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Kdump supported >2014-10-23 14:41:59,366 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Kdump supported. >2014-10-23 14:41:59,379 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20141023144235-5i6f4f.log' >2014-10-23 14:41:59,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Logs at host located at: '/tmp/ovirt-host-deploy-20141023144235-5i6f4f.log'. >2014-10-23 14:41:59,656 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host monique-vds01.tlv.redhat.com reports unique id 44454C4C-5A00-1044-804A-CAC04F32354A >2014-10-23 14:41:59,716 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 44454C4C-5A00-1044-804A-CAC04F32354A to Host monique-vds01.tlv.redhat.com >2014-10-23 14:42:00,242 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Disabling Kdump integration >2014-10-23 14:42:00,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Disabling Kdump integration. >2014-10-23 14:42:00,408 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Setup validation >2014-10-23 14:42:00,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Setup validation. >2014-10-23 14:42:00,450 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Hardware supports virtualization >2014-10-23 14:42:00,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Hardware supports virtualization. >2014-10-23 14:42:00,697 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Transaction setup >2014-10-23 14:42:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Transaction setup. >2014-10-23 14:42:00,734 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Misc configuration >2014-10-23 14:42:00,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 14:42:00,746 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Package installation >2014-10-23 14:42:00,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Package installation. >2014-10-23 14:42:03,382 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp-/127.0.0.1:8702-8) [25aad210] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: POOL >] >2014-10-23 14:42:03,569 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 14:42:03,573 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 497bcf80 >2014-10-23 14:42:03,573 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] VDS monique-vds05.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-10-23 14:42:03,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] START, ResetIrsVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, vdsId = acb7e296-b82a-4376-9193-f26aedc8ef44, ignoreStopFailed = false), log id: 120bcbda >2014-10-23 14:42:03,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] START, SpmStopVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849), log id: 265827ea >2014-10-23 14:42:03,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] SpmStopVDSCommand::Stopping SPM on vds monique-vds05.tlv.redhat.com, pool id f19f3eec-207f-4455-8715-dd11cd054849 >2014-10-23 14:42:03,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] FINISH, SpmStopVDSCommand, log id: 265827ea >2014-10-23 14:42:03,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] FINISH, ResetIrsVDSCommand, log id: 120bcbda >2014-10-23 14:42:03,799 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] FINISH, SetVdsStatusVDSCommand, log id: 497bcf80 >2014-10-23 14:42:03,948 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: f19f3eec-207f-4455-8715-dd11cd054849 value: POOL >] >2014-10-23 14:42:04,065 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-7-thread-6) [25aad210] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 14:42:04,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-6) [25aad210] Correlation ID: 25aad210, Job ID: 61f691d7-882d-4c19-beca-96088c55f0b2, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was switched to Maintenance mode by admin. >2014-10-23 14:42:04,331 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = acb7e296-b82a-4376-9193-f26aedc8ef44 : monique-vds05.tlv.redhat.com >2014-10-23 14:42:04,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Clearing cache of pool: f19f3eec-207f-4455-8715-dd11cd054849 for problematic entities of VDS: monique-vds05.tlv.redhat.com. >2014-10-23 14:42:04,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Removing vds [acb7e296-b82a-4376-9193-f26aedc8ef44] from the domain in maintenance cache >2014-10-23 14:42:04,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-23) Removing host(s) [acb7e296-b82a-4376-9193-f26aedc8ef44] from hosts unseen domain report cache >2014-10-23 14:42:04,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) START, DisconnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, vds_spm_id = 2), log id: 46f84664 >2014-10-23 14:42:06,076 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:42:06,079 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [7c3a6889] Storage Pool f19f3eec-207f-4455-8715-dd11cd054849 - Updating Storage Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63 status from Active to Unknown, reason : null >2014-10-23 14:42:06,129 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7c3a6889] Correlation ID: 7c3a6889, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting status to Non Responsive. >2014-10-23 14:42:06,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [7c3a6889] hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-23 14:42:06,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [7c3a6889] starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-10-23 14:42:06,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 320b2009 >2014-10-23 14:42:06,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] spmStart polling started: taskId = 5a8da048-cc5c-4763-8a33-1628640c6b5f >2014-10-23 14:42:07,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] spmStart polling ended: taskId = 5a8da048-cc5c-4763-8a33-1628640c6b5f task status = finished >2014-10-23 14:42:07,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] spmStart polling ended, spm status: SPM >2014-10-23 14:42:07,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, taskId=5a8da048-cc5c-4763-8a33-1628640c6b5f), log id: 3959face >2014-10-23 14:42:07,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] FINISH, HSMClearTaskVDSCommand, log id: 3959face >2014-10-23 14:42:07,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) [7c3a6889] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e0bf10, log id: 320b2009 >2014-10-23 14:42:07,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [7c3a6889] Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-10-23 14:42:07,668 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7c3a6889] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-10-23 14:42:07,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false), log id: 394d246c >2014-10-23 14:42:07,806 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:42:07,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, DisconnectStoragePoolVDSCommand, log id: 46f84664 >2014-10-23 14:42:07,971 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [133a18eb] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:42:08,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [133a18eb] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 224954eb >2014-10-23 14:42:08,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] -- executeIrsBrokerCommand: Attempting on storage pool f19f3eec-207f-4455-8715-dd11cd054849 >2014-10-23 14:42:08,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 3d64a06f >2014-10-23 14:42:08,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d64a06f >2014-10-23 14:42:08,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 394d246c >2014-10-23 14:42:08,166 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-13) [7c3a6889] Discovered no tasks on Storage Pool DC1 >2014-10-23 14:42:08,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [133a18eb] FINISH, DisconnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 224954eb >2014-10-23 14:42:08,831 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Misc configuration >2014-10-23 14:42:08,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Misc configuration. >2014-10-23 14:42:10,504 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Enrolling certificate >2014-10-23 14:42:10,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Enrolling certificate. >2014-10-23 14:42:11,336 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-2) [857769c] Lock Acquired to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 14:42:11,342 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-2) [857769c] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN, ID: 5b7d55fe-d719-41ee-a168-8a17e422ca15 Type: VdsGroupsAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 14:42:11,534 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] START, RemoveVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 2dcf7cba >2014-10-23 14:42:11,535 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [857769c] vdsManager::disposing >2014-10-23 14:42:11,543 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] FINISH, RemoveVdsVDSCommand, log id: 2dcf7cba >2014-10-23 14:42:11,546 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] START, AddVdsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 2c0ff9cb >2014-10-23 14:42:11,546 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] AddVds - entered , starting logic to add VDS acb7e296-b82a-4376-9193-f26aedc8ef44 >2014-10-23 14:42:11,548 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] AddVds - VDS acb7e296-b82a-4376-9193-f26aedc8ef44 was added, will try to add it to the resource manager >2014-10-23 14:42:11,548 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [857769c] Entered VdsManager constructor >2014-10-23 14:42:11,570 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [857769c] Initialize vdsBroker (monique-vds05.tlv.redhat.com,54,321) >2014-10-23 14:42:11,571 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [857769c] VDS acb7e296-b82a-4376-9193-f26aedc8ef44 was added to the Resource Manager >2014-10-23 14:42:11,572 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [857769c] FINISH, AddVdsVDSCommand, log id: 2c0ff9cb >2014-10-23 14:42:11,598 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [29a925de] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 14:42:11,673 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [29a925de] Lock freed to object EngineLock [exclusiveLocks= key: 6705343f-9285-4942-be95-21ea33bb0125 value: REGISTER_VDS >, sharedLocks= ] >2014-10-23 14:42:11,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [29a925de] Correlation ID: 857769c, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-10-23 14:42:11,770 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-2) [29a925de] Lock freed to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 14:42:11,815 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-7-thread-16) [47a73cc2] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 14:42:11,892 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-16) [47a73cc2] Running command: SetupNetworksCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-10-23 14:42:11,893 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-7-thread-16) [47a73cc2] No changes were detected in setup networks for host monique-vds05.tlv.redhat.com (ID: acb7e296-b82a-4376-9193-f26aedc8ef44) >2014-10-23 14:42:11,994 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:42:11,994 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:42:11,995 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-8) [6c864a4b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 14:42:12,020 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:42:12,020 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-8) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:42:12,026 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [6c864a4b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 14:42:12,050 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Transaction commit >2014-10-23 14:42:12,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Transaction commit. >2014-10-23 14:42:12,079 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [6c864a4b] Correlation ID: 6c864a4b, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com configuration was updated by admin. >2014-10-23 14:42:13,452 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Closing up >2014-10-23 14:42:13,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Closing up. >2014-10-23 14:42:14,579 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Initializing Host: monique-vds05.tlv.redhat.com >2014-10-23 14:42:15,178 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stopping libvirtd >2014-10-23 14:42:15,195 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stopping libvirtd. >2014-10-23 14:42:15,929 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Starting vdsm >2014-10-23 14:42:15,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Starting vdsm. >2014-10-23 14:42:16,458 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-2) [41331e5d] Lock Acquired to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 14:42:16,525 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Running command: ActivateVdsCommand internal: false. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 14:42:16,526 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Before acquiring lock in order to prevent monitoring for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-10-23 14:42:16,527 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Lock acquired, from now a monitoring of host will be skipped for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-10-23 14:42:16,571 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] START, SetVdsStatusVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3a4176d5 >2014-10-23 14:42:16,602 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] FINISH, SetVdsStatusVDSCommand, log id: 3a4176d5 >2014-10-23 14:42:16,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Activate finished. Lock released. Monitoring can run now for host monique-vds05.tlv.redhat.com from data-center DC2 >2014-10-23 14:42:16,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Correlation ID: 41331e5d, Job ID: 6c5aac21-5128-40b0-8563-909b32147d8d, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds05.tlv.redhat.com was activated by admin. >2014-10-23 14:42:16,793 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-2) [41331e5d] Lock freed to object EngineLock [exclusiveLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS >, sharedLocks= ] >2014-10-23 14:42:17,614 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 14:42:17,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vds=Host[monique-vds05.tlv.redhat.com,acb7e296-b82a-4376-9193-f26aedc8ef44]), log id: 5dd4bdd9 >2014-10-23 14:42:17,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, GetHardwareInfoVDSCommand, log id: 5dd4bdd9 >2014-10-23 14:42:17,926 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [115a6b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 14:42:18,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [115a6b0] START, GetHardwareInfoVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vds=Host[monique-vds05.tlv.redhat.com,acb7e296-b82a-4376-9193-f26aedc8ef44]), log id: 17b4b270 >2014-10-23 14:42:18,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [115a6b0] FINISH, GetHardwareInfoVDSCommand, log id: 17b4b270 >2014-10-23 14:42:18,120 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-41) [61e9761b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 6705343f-9285-4942-be95-21ea33bb0125 Type: StoragePool >2014-10-23 14:42:18,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-41) [61e9761b] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 70d137e3 >2014-10-23 14:42:18,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-41) [61e9761b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 70d137e3 >2014-10-23 14:42:18,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [61e9761b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds05.tlv.redhat.com was set to Up. >2014-10-23 14:42:18,298 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [61e9761b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds05.tlv.redhat.com. >2014-10-23 14:42:18,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [910c607] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: acb7e296-b82a-4376-9193-f26aedc8ef44 Type: VDS >2014-10-23 14:42:20,536 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Pre-termination >2014-10-23 14:42:20,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Pre-termination. >2014-10-23 14:42:20,594 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023144220-monique-vds01.tlv.redhat.com-7cfeb33c.log' >2014-10-23 14:42:20,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20141023144220-monique-vds01.tlv.redhat.com-7cfeb33c.log'. >2014-10-23 14:42:21,124 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation monique-vds01.tlv.redhat.com: Stage: Termination >2014-10-23 14:42:21,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 7cfeb33c, Call Stack: null, Custom Event ID: -1, Message: Installing Host monique-vds01.tlv.redhat.com. Stage: Termination. >2014-10-23 14:42:21,272 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 14:42:21,273 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 14:42:21,274 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 14:42:21,274 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (org.ovirt.thread.pool-7-thread-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:917) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:913) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:405) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:394) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-23 14:42:21,276 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-10-23 14:42:21,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Command PollVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-10-23 14:42:21,782 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 14:42:21,899 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Engine managed to communicate with VDSM agent on host monique-vds01.tlv.redhat.com >2014-10-23 14:42:22,625 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] The management network rhevm is already configured on host monique-vds01.tlv.redhat.com >2014-10-23 14:42:22,629 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] START, SetVdsStatusVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 38711ea9 >2014-10-23 14:42:22,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] FINISH, SetVdsStatusVDSCommand, log id: 38711ea9 >2014-10-23 14:42:22,633 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] After Installation host monique-vds01.tlv.redhat.com, VDS >2014-10-23 14:42:22,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Correlation ID: 7cfeb33c, Job ID: 017858cd-a613-4228-94fa-df04f422b109, Call Stack: null, Custom Event ID: -1, Message: Host monique-vds01.tlv.redhat.com installed >2014-10-23 14:42:22,647 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-49) [7cfeb33c] Lock freed to object EngineLock [exclusiveLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS >, sharedLocks= ] >2014-10-23 14:42:24,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, vds=Host[monique-vds01.tlv.redhat.com,38db43ff-c477-42e3-8fa2-780b6bf164cd]), log id: 1f8f6a8b >2014-10-23 14:42:24,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, GetHardwareInfoVDSCommand, log id: 1f8f6a8b >2014-10-23 14:42:24,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [28fb6477] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 38db43ff-c477-42e3-8fa2-780b6bf164cd Type: VDS >2014-10-23 14:42:24,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [28fb6477] START, GetHardwareInfoVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, vds=Host[monique-vds01.tlv.redhat.com,38db43ff-c477-42e3-8fa2-780b6bf164cd]), log id: 7ec1f089 >2014-10-23 14:42:24,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [28fb6477] FINISH, GetHardwareInfoVDSCommand, log id: 7ec1f089 >2014-10-23 14:42:25,026 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-2) [790b5ce3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: d4869232-1f76-44ed-9c03-2755258c43d5 Type: StoragePool >2014-10-23 14:42:25,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-2) [790b5ce3] START, SetMOMPolicyParametersVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd), log id: 1b38793d >2014-10-23 14:42:25,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-2) [790b5ce3] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1b38793d >2014-10-23 14:42:25,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [790b5ce3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host monique-vds01.tlv.redhat.com was set to Up. >2014-10-23 14:42:25,159 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [790b5ce3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host monique-vds01.tlv.redhat.com. >2014-10-23 14:42:25,182 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [70b5c7ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 38db43ff-c477-42e3-8fa2-780b6bf164cd Type: VDS >2014-10-23 14:42:25,196 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-2) [65fa42d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 38db43ff-c477-42e3-8fa2-780b6bf164cd Type: VDS >2014-10-23 14:44:22,673 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [300292ae] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 14:44:22,765 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [300292ae] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:22,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [300292ae] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 37042ad1 >2014-10-23 14:44:22,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [300292ae] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 37042ad1 >2014-10-23 14:44:22,881 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [300292ae] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 14:44:23,084 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [4937779] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:23,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 44b2e4ac-c110-463f-99c2-4ee01f6dd52e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2b03508d >2014-10-23 14:44:23,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] FINISH, ConnectStorageServerVDSCommand, return: {44b2e4ac-c110-463f-99c2-4ee01f6dd52e=0}, log id: 2b03508d >2014-10-23 14:44:23,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] START, CreateStorageDomainVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storageDomain=StorageDomainStatic[SD2, b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3), log id: 3cf67920 >2014-10-23 14:44:23,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] FINISH, CreateStorageDomainVDSCommand, log id: 3cf67920 >2014-10-23 14:44:23,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] START, GetStorageDomainStatsVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storageDomainId=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc), log id: 335f12f4 >2014-10-23 14:44:23,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-1) [4937779] FINISH, GetStorageDomainStatsVDSCommand, return: StorageDomain[, b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc], log id: 335f12f4 >2014-10-23 14:44:23,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [4937779] Correlation ID: 4937779, Job ID: e28f091a-d220-4250-b36b-d461cf75644d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD2 was added by admin >2014-10-23 14:44:23,932 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [51e10f25] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:23,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [51e10f25] START, DisconnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 44b2e4ac-c110-463f-99c2-4ee01f6dd52e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6717ebda >2014-10-23 14:44:24,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [51e10f25] FINISH, DisconnectStorageServerVDSCommand, return: {44b2e4ac-c110-463f-99c2-4ee01f6dd52e=0}, log id: 6717ebda >2014-10-23 14:44:24,307 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-6) [2e2cfa0c] Lock Acquired to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= ] >2014-10-23 14:44:24,354 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-6) [2e2cfa0c] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:24,409 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-6) [2c97a8c3] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 6705343f-9285-4942-be95-21ea33bb0125 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 14:44:24,666 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:24,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 44b2e4ac-c110-463f-99c2-4ee01f6dd52e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4943acb3 >2014-10-23 14:44:24,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] FINISH, ConnectStorageServerVDSCommand, return: {44b2e4ac-c110-463f-99c2-4ee01f6dd52e=0}, log id: 4943acb3 >2014-10-23 14:44:24,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] START, CreateStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId=6705343f-9285-4942-be95-21ea33bb0125, storagePoolName=DC2, masterDomainId=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, domainsIdList=[b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc], masterVersion=1), log id: 42a98afa >2014-10-23 14:44:48,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] FINISH, CreateStoragePoolVDSCommand, log id: 42a98afa >2014-10-23 14:44:48,358 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-6) [7f61119a] Lock freed to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= ] >2014-10-23 14:44:48,362 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6340d2f3] Lock Acquired to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= ] >2014-10-23 14:44:48,369 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6340d2f3] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:48,376 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6340d2f3] Lock freed to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= ] >2014-10-23 14:44:48,377 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6340d2f3] ActivateStorage Domain. Before Connect all hosts to pool. Time:10/23/14 2:44 PM >2014-10-23 14:44:48,383 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-27) [7b68b3cf] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:48,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-27) [7b68b3cf] START, ConnectStorageServerVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 44b2e4ac-c110-463f-99c2-4ee01f6dd52e, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs3, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 8ca42e5 >2014-10-23 14:44:48,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-27) [7b68b3cf] FINISH, ConnectStorageServerVDSCommand, return: {44b2e4ac-c110-463f-99c2-4ee01f6dd52e=0}, log id: 8ca42e5 >2014-10-23 14:44:48,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [6340d2f3] START, ActivateStorageDomainVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc), log id: 1869f008 >2014-10-23 14:44:48,463 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-6) [138f181d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 6705343f-9285-4942-be95-21ea33bb0125 Type: StoragePool >2014-10-23 14:44:48,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [138f181d] Correlation ID: 138f181d, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-10-23 14:44:48,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-6) [138f181d] hostFromVds::selectedVds - monique-vds05.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC2 >2014-10-23 14:44:48,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vdsId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, masterVersion = 1), log id: 18443541 >2014-10-23 14:44:48,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] FINISH, ConnectStoragePoolVDSCommand, log id: 18443541 >2014-10-23 14:44:48,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-6) [138f181d] hostFromVds::Connected host to pool - selectedVds - monique-vds05.tlv.redhat.com, spmStatus Free, storage pool DC2 >2014-10-23 14:44:48,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-6) [138f181d] starting spm on vds monique-vds05.tlv.redhat.com, storage pool DC2, prevId -1, LVER -1 >2014-10-23 14:44:48,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] START, SpmStartVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76029ba1 >2014-10-23 14:44:48,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] spmStart polling started: taskId = 183ec94d-157f-4363-b439-1a1760378b09 >2014-10-23 14:44:55,721 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [a4aa688] Lock Acquired to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 14:44:55,736 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [a4aa688] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:55,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [a4aa688] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4b6c9b5f >2014-10-23 14:44:55,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [a4aa688] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4b6c9b5f >2014-10-23 14:44:55,920 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [a4aa688] Lock freed to object EngineLock [exclusiveLocks= key: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-10-23 14:44:56,108 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:56,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 652ffe9d-bf63-4bdb-aa25-707055b2106c, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f9e20b8 >2014-10-23 14:44:56,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] FINISH, ConnectStorageServerVDSCommand, return: {652ffe9d-bf63-4bdb-aa25-707055b2106c=0}, log id: 4f9e20b8 >2014-10-23 14:44:56,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] START, CreateStorageDomainVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storageDomain=StorageDomainStatic[SD3, 6de9807c-ffcd-4be1-a76e-b5eaefbb7627], args=lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4), log id: 4d2ed222 >2014-10-23 14:44:56,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] FINISH, CreateStorageDomainVDSCommand, log id: 4d2ed222 >2014-10-23 14:44:56,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] START, GetStorageDomainStatsVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storageDomainId=6de9807c-ffcd-4be1-a76e-b5eaefbb7627), log id: 311189e0 >2014-10-23 14:44:56,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-5) [964c0bd] FINISH, GetStorageDomainStatsVDSCommand, return: StorageDomain[, 6de9807c-ffcd-4be1-a76e-b5eaefbb7627], log id: 311189e0 >2014-10-23 14:44:56,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [964c0bd] Correlation ID: 964c0bd, Job ID: 1d8fa0b6-b807-4a38-b0c2-22a8f400c326, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD3 was added by admin >2014-10-23 14:44:57,030 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-11) [3bec636b] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:57,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [3bec636b] START, DisconnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 652ffe9d-bf63-4bdb-aa25-707055b2106c, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3fedf2de >2014-10-23 14:44:57,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [3bec636b] FINISH, DisconnectStorageServerVDSCommand, return: {652ffe9d-bf63-4bdb-aa25-707055b2106c=0}, log id: 3fedf2de >2014-10-23 14:44:57,446 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-2) [1362a3f5] Lock Acquired to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= ] >2014-10-23 14:44:57,478 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-2) [1362a3f5] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:57,597 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-2) [61da5de1] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: d4869232-1f76-44ed-9c03-2755258c43d5 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN >2014-10-23 14:44:57,758 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-2) [58908586] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:44:57,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58908586] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 652ffe9d-bf63-4bdb-aa25-707055b2106c, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 762195db >2014-10-23 14:44:57,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [58908586] FINISH, ConnectStorageServerVDSCommand, return: {652ffe9d-bf63-4bdb-aa25-707055b2106c=0}, log id: 762195db >2014-10-23 14:44:57,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-2) [58908586] START, CreateStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId=d4869232-1f76-44ed-9c03-2755258c43d5, storagePoolName=DC3, masterDomainId=6de9807c-ffcd-4be1-a76e-b5eaefbb7627, domainsIdList=[6de9807c-ffcd-4be1-a76e-b5eaefbb7627], masterVersion=1), log id: 2fa53184 >2014-10-23 14:45:11,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] spmStart polling ended: taskId = 183ec94d-157f-4363-b439-1a1760378b09 task status = finished >2014-10-23 14:45:11,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] spmStart polling ended, spm status: SPM >2014-10-23 14:45:11,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=183ec94d-157f-4363-b439-1a1760378b09), log id: 191277b0 >2014-10-23 14:45:11,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] FINISH, HSMClearTaskVDSCommand, log id: 191277b0 >2014-10-23 14:45:11,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@593b79b, log id: 76029ba1 >2014-10-23 14:45:11,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-6) [138f181d] Initialize Irs proxy from vds: monique-vds05.tlv.redhat.com >2014-10-23 14:45:11,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [138f181d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds05.tlv.redhat.com (Address: monique-vds05.tlv.redhat.com). >2014-10-23 14:45:11,244 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds05.tlv.redhat.com/10.35.2.49 >2014-10-23 14:45:11,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [138f181d] FINISH, ActivateStorageDomainVDSCommand, return: true, log id: 1869f008 >2014-10-23 14:45:11,360 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [138f181d] ActivateStorage Domain. After Connect all hosts to pool. Time:10/23/14 2:45 PM >2014-10-23 14:45:11,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) START, ConnectStoragePoolVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, vdsId = acb7e296-b82a-4376-9193-f26aedc8ef44, storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, masterVersion = 1), log id: 3946deda >2014-10-23 14:45:11,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 3946deda >2014-10-23 14:45:11,624 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [138f181d] ActivateStorage Domain. After change storage pool status in vds. Time:10/23/14 2:45 PM >2014-10-23 14:45:11,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [138f181d] Correlation ID: 6340d2f3, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD2 (Data Center DC2) was activated by admin >2014-10-23 14:45:11,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [138f181d] Correlation ID: 2c97a8c3, Job ID: da0dd25f-a846-4a93-a815-ae388091ac11, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center DC2 by admin >2014-10-23 14:45:11,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [138f181d] Correlation ID: 2e2cfa0c, Job ID: da0dd25f-a846-4a93-a815-ae388091ac11, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD2 was attached to Data Center DC2 by admin >2014-10-23 14:45:11,681 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ajp-/127.0.0.1:8702-6) [138f181d] Trying to release exclusive lock which does not exist, lock key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfcSTORAGE >2014-10-23 14:45:11,682 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-6) [138f181d] Lock freed to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= ] >2014-10-23 14:45:21,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-2) [58908586] FINISH, CreateStoragePoolVDSCommand, log id: 2fa53184 >2014-10-23 14:45:21,402 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-2) [58908586] Lock freed to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= ] >2014-10-23 14:45:21,405 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6c3b8108] Lock Acquired to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= ] >2014-10-23 14:45:21,420 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6c3b8108] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:45:21,426 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6c3b8108] Lock freed to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= ] >2014-10-23 14:45:21,426 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6c3b8108] ActivateStorage Domain. Before Connect all hosts to pool. Time:10/23/14 2:45 PM >2014-10-23 14:45:21,433 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-15) [66001471] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN >2014-10-23 14:45:21,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-15) [66001471] START, ConnectStorageServerVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 652ffe9d-bf63-4bdb-aa25-707055b2106c, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs4, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e9fd181 >2014-10-23 14:45:21,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-15) [66001471] FINISH, ConnectStorageServerVDSCommand, return: {652ffe9d-bf63-4bdb-aa25-707055b2106c=0}, log id: 7e9fd181 >2014-10-23 14:45:21,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [6c3b8108] START, ActivateStorageDomainVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627), log id: 6c1cfdcb >2014-10-23 14:45:21,518 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: d4869232-1f76-44ed-9c03-2755258c43d5 Type: StoragePool >2014-10-23 14:45:21,524 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3bcf678e] Correlation ID: 3bcf678e, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-10-23 14:45:21,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-2) [3bcf678e] hostFromVds::selectedVds - monique-vds01.tlv.redhat.com, spmStatus Unknown_Pool, storage pool DC3 >2014-10-23 14:45:21,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, vdsId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, masterVersion = 1), log id: 53467739 >2014-10-23 14:45:22,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] FINISH, ConnectStoragePoolVDSCommand, log id: 53467739 >2014-10-23 14:45:22,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-2) [3bcf678e] hostFromVds::Connected host to pool - selectedVds - monique-vds01.tlv.redhat.com, spmStatus Free, storage pool DC3 >2014-10-23 14:45:22,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-2) [3bcf678e] starting spm on vds monique-vds01.tlv.redhat.com, storage pool DC3, prevId -1, LVER -1 >2014-10-23 14:45:22,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] START, SpmStartVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d00a32 >2014-10-23 14:45:22,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] spmStart polling started: taskId = ccb559eb-1013-4df0-8af8-899c3af5c820 >2014-10-23 14:45:43,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] spmStart polling ended: taskId = ccb559eb-1013-4df0-8af8-899c3af5c820 task status = finished >2014-10-23 14:45:43,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] spmStart polling ended, spm status: SPM >2014-10-23 14:45:43,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, taskId=ccb559eb-1013-4df0-8af8-899c3af5c820), log id: 4d64a97e >2014-10-23 14:45:43,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] FINISH, HSMClearTaskVDSCommand, log id: 4d64a97e >2014-10-23 14:45:43,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@700acb2b, log id: d00a32 >2014-10-23 14:45:43,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp-/127.0.0.1:8702-2) [3bcf678e] Initialize Irs proxy from vds: monique-vds01.tlv.redhat.com >2014-10-23 14:45:43,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3bcf678e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host monique-vds01.tlv.redhat.com (Address: monique-vds01.tlv.redhat.com). >2014-10-23 14:45:43,370 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to monique-vds01.tlv.redhat.com/10.35.2.45 >2014-10-23 14:45:43,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] FINISH, ActivateStorageDomainVDSCommand, return: true, log id: 6c1cfdcb >2014-10-23 14:45:43,491 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] ActivateStorage Domain. After Connect all hosts to pool. Time:10/23/14 2:45 PM >2014-10-23 14:45:43,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-22) START, ConnectStoragePoolVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, vdsId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, masterVersion = 1), log id: 52bfe549 >2014-10-23 14:45:43,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 52bfe549 >2014-10-23 14:45:43,991 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] ActivateStorage Domain. After change storage pool status in vds. Time:10/23/14 2:45 PM >2014-10-23 14:45:43,997 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3bcf678e] Correlation ID: 6c3b8108, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD3 (Data Center DC3) was activated by admin >2014-10-23 14:45:44,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3bcf678e] Correlation ID: 61da5de1, Job ID: 8de2726c-10af-4081-b017-d5c7032ec56d, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center DC3 by admin >2014-10-23 14:45:44,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [3bcf678e] Correlation ID: 1362a3f5, Job ID: 8de2726c-10af-4081-b017-d5c7032ec56d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain SD3 was attached to Data Center DC3 by admin >2014-10-23 14:45:44,016 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ajp-/127.0.0.1:8702-2) [3bcf678e] Trying to release exclusive lock which does not exist, lock key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627STORAGE >2014-10-23 14:45:44,017 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-2) [3bcf678e] Lock freed to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= ] >2014-10-23 14:51:29,586 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:51:29,586 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:51:29,586 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-7) [492ad15e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-10-23 14:51:29,619 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:51:29,619 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-10-23 14:51:29,824 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 14:51:29,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: 492ad15e, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com configuration was updated by admin. >2014-10-23 14:54:02,444 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetStatsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 14:54:02,448 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-10-23 14:54:02,505 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com is not responding. It will stay in Connecting state for a grace period of 80 seconds and after that an attempt to fence the host will be issued. >2014-10-23 14:54:02,506 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) vds::refreshVdsStats Failed getVdsStats, vds = 2c075add-8452-44b3-872d-20db3a5f1c2b : rose07.qa.lab.tlv.redhat.com, error = VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 14:54:02,506 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand.executeVdsBrokerCommand(GetStatsVDSCommand.java:24) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:698) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:652) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-10-23 14:54:02,521 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 2c075add-8452-44b3-872d-20db3a5f1c2b : rose07.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 14:54:05,696 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:54:06,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 1a49dc01 >2014-10-23 14:54:06,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, GetHardwareInfoVDSCommand, log id: 1a49dc01 >2014-10-23 14:54:06,356 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [3c87ad4a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:54:06,377 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [6b75f5a2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:54:06,386 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [38ed8b49] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:54:06,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [38ed8b49] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c83414e >2014-10-23 14:54:06,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [38ed8b49] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 6c83414e >2014-10-23 14:54:06,694 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [38ed8b49] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 14:54:06,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 3a5f8be4 >2014-10-23 14:54:06,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3a5f8be4 >2014-10-23 14:54:13,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) Command SpmStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 14:54:13,990 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-30) [4d78cb0b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:54:13,992 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-30) [4d78cb0b] Storage Pool f19f3eec-207f-4455-8715-dd11cd054849 - Updating Storage Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63 status from Active to Unknown, reason : null >2014-10-23 14:54:14,059 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [4d78cb0b] Correlation ID: 4d78cb0b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center DC1. Setting Data Center status to Non Responsive (On host rose07.qa.lab.tlv.redhat.com, Error: Network error during communication with the Host.). >2014-10-23 14:54:14,066 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:54:24,186 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-67) Storage Pool f19f3eec-207f-4455-8715-dd11cd054849 - Updating Storage Domain e22453d1-773c-4f02-8222-a2c0f2aa5f63 status from Active to Unknown, reason : no reporting hosts >2014-10-23 14:54:24,190 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:54:24,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-10-23 14:54:24,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-67) HostName = rose07.qa.lab.tlv.redhat.com >2014-10-23 14:54:24,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command HSMGetAllTasksStatusesVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-10-23 14:57:07,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-7-thread-46) Command GetStatsVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-10-23 14:57:07,080 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-46) Could not get Host statistics for Host rose07.qa.lab.tlv.redhat.com, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2031) [bll.jar:] > at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:639) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] > >2014-10-23 14:57:07,122 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com reports about one of the Active Storage Domains as Problematic. >2014-10-23 14:57:07,212 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:57:07,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 9f52307 >2014-10-23 14:57:07,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] VDS rose07.qa.lab.tlv.redhat.com is spm and moved from up calling resetIrs. >2014-10-23 14:57:07,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] START, ResetIrsVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, ignoreStopFailed = false), log id: 7e603c9a >2014-10-23 14:57:07,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] START, SpmStopVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849), log id: 357984d2 >2014-10-23 14:57:07,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] SpmStopVDSCommand:: vds rose07.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id f19f3eec-207f-4455-8715-dd11cd054849 >2014-10-23 14:57:07,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] FINISH, SpmStopVDSCommand, log id: 357984d2 >2014-10-23 14:57:07,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] FINISH, ResetIrsVDSCommand, log id: 7e603c9a >2014-10-23 14:57:07,282 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [592dea5a] FINISH, SetVdsStatusVDSCommand, log id: 9f52307 >2014-10-23 14:57:07,302 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [592dea5a] Correlation ID: 592dea5a, Job ID: cb1a8fc0-c23f-46d7-b0e1-154816259be5, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DC1. Setting Host state to Non-Operational. >2014-10-23 14:57:07,366 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [592dea5a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 14:57:07,390 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [592dea5a] Correlation ID: 6b75f5a2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rose07.qa.lab.tlv.redhat.com to Storage Pool DC1 >2014-10-23 14:57:07,502 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [105ea1be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:57:07,509 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [105ea1be] Host 2c075add-8452-44b3-872d-20db3a5f1c2b : rose07.qa.lab.tlv.redhat.com is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-10-23 14:57:49,087 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-7) [702a63ae] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 14:57:49,126 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-14) [702a63ae] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 14:57:49,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-14) [702a63ae] START, ResetIrsVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, ignoreStopFailed = false), log id: 5cd34d43 >2014-10-23 14:57:49,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-7-thread-14) [702a63ae] FINISH, ResetIrsVDSCommand, log id: 5cd34d43 >2014-10-23 14:57:49,212 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:57:49,269 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] START, UpdateVdsVMsClearedVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 5ecae672 >2014-10-23 14:57:49,308 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 5ecae672 >2014-10-23 14:57:49,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] Correlation ID: 2db0ee56, Call Stack: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host rose07.qa.lab.tlv.redhat.com were changed to 'Down' by admin >2014-10-23 14:57:49,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] Correlation ID: 702a63ae, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host rose07.qa.lab.tlv.redhat.com was started. >2014-10-23 14:57:49,401 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-7-thread-14) [2db0ee56] Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS_FENCE >, sharedLocks= ] >2014-10-23 14:57:55,197 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-5) [3a16e243] Lock Acquired to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 14:57:55,326 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-10-23 14:57:55,327 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Before acquiring lock in order to prevent monitoring for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 14:57:55,327 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Lock acquired, from now a monitoring of host will be skipped for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 14:57:55,362 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] START, SetVdsStatusVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a7583bd >2014-10-23 14:57:55,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] FINISH, SetVdsStatusVDSCommand, log id: 7a7583bd >2014-10-23 14:57:55,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] START, SetHaMaintenanceModeVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 60b0f24d >2014-10-23 14:57:55,523 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:57:55,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] FINISH, SetHaMaintenanceModeVDSCommand, log id: 60b0f24d >2014-10-23 14:57:55,699 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Activate finished. Lock released. Monitoring can run now for host rose07.qa.lab.tlv.redhat.com from data-center DC1 >2014-10-23 14:57:55,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Correlation ID: 3a16e243, Job ID: 365969e0-6c85-4a83-b651-d4faedf7be2c, Call Stack: null, Custom Event ID: -1, Message: Host rose07.qa.lab.tlv.redhat.com was activated by admin. >2014-10-23 14:57:55,709 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-7-thread-9) [3a16e243] Lock freed to object EngineLock [exclusiveLocks= key: 2c075add-8452-44b3-872d-20db3a5f1c2b value: VDS >, sharedLocks= ] >2014-10-23 14:57:56,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) START, GetHardwareInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vds=Host[rose07.qa.lab.tlv.redhat.com,2c075add-8452-44b3-872d-20db3a5f1c2b]), log id: 39f82945 >2014-10-23 14:57:56,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, GetHardwareInfoVDSCommand, log id: 39f82945 >2014-10-23 14:57:56,362 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [6d9f71f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:57:56,385 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [1b4b6d0e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:57:56,395 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f19f3eec-207f-4455-8715-dd11cd054849 Type: StoragePool >2014-10-23 14:57:56,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] START, ConnectStorageServerVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, storageType = NFS, connectionList = [{ id: c0b0bb71-688b-4e91-9422-a68ceb7895c9, connection: lion.qa.lab.tlv.redhat.com:/export/slitmano/nfs2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50b0964c >2014-10-23 14:57:56,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] FINISH, ConnectStorageServerVDSCommand, return: {c0b0bb71-688b-4e91-9422-a68ceb7895c9=0}, log id: 50b0964c >2014-10-23 14:57:56,445 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] Host rose07.qa.lab.tlv.redhat.com storage connection was succeeded >2014-10-23 14:57:56,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, vdsId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, masterVersion = 1), log id: 3addcf69 >2014-10-23 14:57:56,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-7-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 3addcf69 >2014-10-23 14:58:21,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] START, SetMOMPolicyParametersVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 2c1c5261 >2014-10-23 14:58:22,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2c1c5261 >2014-10-23 14:58:22,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rose07.qa.lab.tlv.redhat.com was set to Up. >2014-10-23 14:58:22,108 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [3baeeeb9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rose07.qa.lab.tlv.redhat.com. >2014-10-23 14:58:22,159 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [7c864a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 2c075add-8452-44b3-872d-20db3a5f1c2b Type: VDS >2014-10-23 14:58:24,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - rose07.qa.lab.tlv.redhat.com, spmStatus Free, storage pool DC1 >2014-10-23 14:58:25,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) starting spm on vds rose07.qa.lab.tlv.redhat.com, storage pool DC1, prevId -1, LVER -1 >2014-10-23 14:58:25,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-44) START, SpmStartVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6620bcc6 >2014-10-23 14:58:25,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-44) spmStart polling started: taskId = 0c77d735-54bb-4d42-abd8-2873a8193f86 >2014-10-23 14:58:26,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-44) spmStart polling ended: taskId = 0c77d735-54bb-4d42-abd8-2873a8193f86 task status = finished >2014-10-23 14:58:26,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-44) spmStart polling ended, spm status: SPM >2014-10-23 14:58:26,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-44) START, HSMClearTaskVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b, taskId=0c77d735-54bb-4d42-abd8-2873a8193f86), log id: 4b4a1897 >2014-10-23 14:58:26,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, HSMClearTaskVDSCommand, log id: 4b4a1897 >2014-10-23 14:58:26,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e9a2dd9, log id: 6620bcc6 >2014-10-23 14:58:26,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) Initialize Irs proxy from vds: rose07.qa.lab.tlv.redhat.com >2014-10-23 14:58:26,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rose07.qa.lab.tlv.redhat.com (Address: rose07.qa.lab.tlv.redhat.com). >2014-10-23 14:58:26,236 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to rose07.qa.lab.tlv.redhat.com/10.35.97.75 >2014-10-23 14:58:26,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = f19f3eec-207f-4455-8715-dd11cd054849, ignoreFailoverLimit = false), log id: 7bcc9c1d >2014-10-23 14:58:26,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-43) -- executeIrsBrokerCommand: Attempting on storage pool f19f3eec-207f-4455-8715-dd11cd054849 >2014-10-23 14:58:26,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-43) START, HSMGetAllTasksInfoVDSCommand(HostName = rose07.qa.lab.tlv.redhat.com, HostId = 2c075add-8452-44b3-872d-20db3a5f1c2b), log id: 2f053987 >2014-10-23 14:58:26,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f053987 >2014-10-23 14:58:26,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-7-thread-43) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bcc9c1d >2014-10-23 14:58:26,387 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-43) Discovered no tasks on Storage Pool DC1 >2014-10-23 15:14:58,742 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-10-23 15:15:00,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-10-23 15:15:00,690 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [249a72aa] Lock Acquired to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= key: 6705343f-9285-4942-be95-21ea33bb0125 value: OVF_UPDATE >] >2014-10-23 15:15:00,703 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [249a72aa] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:00,707 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [6d020e16] Running command: CreateOvfStoresForStorageDomainCommand internal: true. >2014-10-23 15:15:00,708 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [681e5df] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:00,981 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-10) [42c50544] Running command: AddDiskCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 15:15:01,051 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-10) [27d46590] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:01,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [27d46590] START, CreateImageVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = 8a376036-bcdf-4cb2-a32b-689200d9b9d1, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = d7a67def-9bd7-431d-9ab3-ce9631f5cd22, newImageDescription = OVF_STORE), log id: 4800746d >2014-10-23 15:15:01,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [27d46590] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 15:15:01,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [27d46590] FINISH, CreateImageVDSCommand, return: d7a67def-9bd7-431d-9ab3-ce9631f5cd22, log id: 4800746d >2014-10-23 15:15:01,217 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-10) [27d46590] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 >2014-10-23 15:15:01,217 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [27d46590] CommandMultiAsyncTasks::AttachTask: Attaching task 73304f9f-fa11-40bf-8047-64f15945c5c7 to command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3. >2014-10-23 15:15:01,232 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [27d46590] Adding task 73304f9f-fa11-40bf-8047-64f15945c5c7 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:01,262 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [27d46590] Correlation ID: 42c50544, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 15:15:01,269 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [7d31512c] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:01,321 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-10) [2062a491] Running command: AddDiskCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 15:15:01,375 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-10) [860ebd3] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:01,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [860ebd3] START, CreateImageVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = dbc1563b-a64c-480e-b70b-aa8f76f95a5c, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 5aa99f91-57c0-4ded-a828-4ec7a49a8c87, newImageDescription = OVF_STORE), log id: 7ad84734 >2014-10-23 15:15:01,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [860ebd3] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 15:15:04,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [860ebd3] FINISH, CreateImageVDSCommand, return: 5aa99f91-57c0-4ded-a828-4ec7a49a8c87, log id: 7ad84734 >2014-10-23 15:15:04,952 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [860ebd3] CommandMultiAsyncTasks::AttachTask: Attaching task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb to command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3. >2014-10-23 15:15:04,967 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [860ebd3] Adding task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:04,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [860ebd3] Correlation ID: 2062a491, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 15:15:04,987 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [860ebd3] BaseAsyncTask::startPollingTask: Starting to poll task 73304f9f-fa11-40bf-8047-64f15945c5c7. >2014-10-23 15:15:04,987 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [860ebd3] BaseAsyncTask::startPollingTask: Starting to poll task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb. >2014-10-23 15:15:04,987 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [860ebd3] Lock freed to object EngineLock [exclusiveLocks= key: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc value: STORAGE >, sharedLocks= key: 6705343f-9285-4942-be95-21ea33bb0125 value: OVF_UPDATE >] >2014-10-23 15:15:05,008 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [6fa9eb0f] Lock Acquired to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= key: d4869232-1f76-44ed-9c03-2755258c43d5 value: OVF_UPDATE >] >2014-10-23 15:15:05,010 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [6fa9eb0f] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:05,011 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [13b733dc] Running command: CreateOvfStoresForStorageDomainCommand internal: true. >2014-10-23 15:15:05,013 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [60adbaa0] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:05,055 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-10) [59c68e6c] Running command: AddDiskCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 15:15:05,096 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-10) [5b401df6] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:05,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [5b401df6] START, CreateImageVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = 62a19a8f-7edd-45b6-9aaa-d585b545509f, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 887d3915-35d1-48b1-bdd4-5f25751e050d, newImageDescription = OVF_STORE), log id: 2b7fb52 >2014-10-23 15:15:05,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [5b401df6] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 15:15:05,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [5b401df6] FINISH, CreateImageVDSCommand, return: 887d3915-35d1-48b1-bdd4-5f25751e050d, log id: 2b7fb52 >2014-10-23 15:15:05,199 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-10) [5b401df6] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command a252321e-f190-4295-a622-61466953cfc7 >2014-10-23 15:15:05,199 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [5b401df6] CommandMultiAsyncTasks::AttachTask: Attaching task bbdbef02-748b-480f-970e-ebfe06150fa2 to command a252321e-f190-4295-a622-61466953cfc7. >2014-10-23 15:15:05,213 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [5b401df6] Adding task bbdbef02-748b-480f-970e-ebfe06150fa2 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:05,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5b401df6] Correlation ID: 59c68e6c, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 15:15:05,231 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [566403d1] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:05,257 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-10) [5ba696d6] Running command: AddDiskCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: StorageAction group CREATE_DISK with role type USER >2014-10-23 15:15:05,282 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-10) [64ed1769] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:05,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [64ed1769] START, CreateImageVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = afebca28-a0c7-47d5-9eea-6019e6dc038e, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 28d85b18-cf5e-4af2-95c7-3075483fa61e, newImageDescription = OVF_STORE), log id: 50e2c229 >2014-10-23 15:15:05,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [64ed1769] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-10-23 15:15:05,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-10) [64ed1769] FINISH, CreateImageVDSCommand, return: 28d85b18-cf5e-4af2-95c7-3075483fa61e, log id: 50e2c229 >2014-10-23 15:15:05,709 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-10) [64ed1769] CommandMultiAsyncTasks::AttachTask: Attaching task b7f10dbd-a8ae-4e27-ab57-488370326327 to command a252321e-f190-4295-a622-61466953cfc7. >2014-10-23 15:15:05,769 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) [64ed1769] Adding task b7f10dbd-a8ae-4e27-ab57-488370326327 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:05,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [64ed1769] Correlation ID: 5ba696d6, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by <UNKNOWN>. >2014-10-23 15:15:05,846 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [64ed1769] BaseAsyncTask::startPollingTask: Starting to poll task bbdbef02-748b-480f-970e-ebfe06150fa2. >2014-10-23 15:15:05,846 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-10) [64ed1769] BaseAsyncTask::startPollingTask: Starting to poll task b7f10dbd-a8ae-4e27-ab57-488370326327. >2014-10-23 15:15:05,847 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-10) [64ed1769] Lock freed to object EngineLock [exclusiveLocks= key: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 value: STORAGE >, sharedLocks= key: d4869232-1f76-44ed-9c03-2755258c43d5 value: OVF_UPDATE >] >2014-10-23 15:15:11,031 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-32) Polling and updating Async Tasks: 4 tasks, 4 tasks to poll now >2014-10-23 15:15:11,081 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) SPMAsyncTask::PollTask: Polling task 73304f9f-fa11-40bf-8047-64f15945c5c7 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) SPMAsyncTask::PollTask: Polling task b7f10dbd-a8ae-4e27-ab57-488370326327 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) BaseAsyncTask::onTaskEndSuccess: Task b7f10dbd-a8ae-4e27-ab57-488370326327 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-32) Task with DB Task ID b9f219d5-8ee2-4d2f-95a6-487ffbae6888 and VDSM Task ID bbdbef02-748b-480f-970e-ebfe06150fa2 is in state Polling. End action for command a252321e-f190-4295-a622-61466953cfc7 will proceed when all the entitys tasks are completed. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) SPMAsyncTask::PollTask: Polling task bbdbef02-748b-480f-970e-ebfe06150fa2 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) BaseAsyncTask::onTaskEndSuccess: Task bbdbef02-748b-480f-970e-ebfe06150fa2 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-32) CommandAsyncTask::endActionIfNecessary: All tasks of command a252321e-f190-4295-a622-61466953cfc7 has ended -> executing endAction >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-32) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: a252321e-f190-4295-a622-61466953cfc7): calling endAction . >2014-10-23 15:15:11,082 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-32) SPMAsyncTask::PollTask: Polling task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-10-23 15:15:11,083 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-32) Finished polling Tasks, will poll again in 10 seconds. >2014-10-23 15:15:11,083 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-31) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction CreateOvfStoresForStorageDomain, executionIndex: 0 >2014-10-23 15:15:11,093 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] Ending command successfully: org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand >2014-10-23 15:15:11,100 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-31) [5b401df6] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 15:15:11,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-31) [5b401df6] START, GetImageInfoVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = 62a19a8f-7edd-45b6-9aaa-d585b545509f, imageId = 887d3915-35d1-48b1-bdd4-5f25751e050d), log id: 3f0497a8 >2014-10-23 15:15:11,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-31) [5b401df6] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@3a6f0b8b, log id: 3f0497a8 >2014-10-23 15:15:11,190 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-31) [64ed1769] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 15:15:11,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-31) [64ed1769] START, GetImageInfoVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = afebca28-a0c7-47d5-9eea-6019e6dc038e, imageId = 28d85b18-cf5e-4af2-95c7-3075483fa61e), log id: 2a8636e8 >2014-10-23 15:15:11,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-31) [64ed1769] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@ab85aecf, log id: 2a8636e8 >2014-10-23 15:15:11,288 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] START, SetVolumeDescriptionVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = afebca28-a0c7-47d5-9eea-6019e6dc038e, imageId = 28d85b18-cf5e-4af2-95c7-3075483fa61e), log id: 2f0c553e >2014-10-23 15:15:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] -- setVolumeDescription parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGroupGUID=afebca28-a0c7-47d5-9eea-6019e6dc038e > volUUID=28d85b18-cf5e-4af2-95c7-3075483fa61e > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"6de9807c-ffcd-4be1-a76e-b5eaefbb7627"}],"Last Updated":null} > >2014-10-23 15:15:11,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [13b733dc] FINISH, SetVolumeDescriptionVDSCommand, log id: 2f0c553e >2014-10-23 15:15:11,486 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS_EXECUTION >] >2014-10-23 15:15:11,497 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] Running command: UploadStreamCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:11,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] START, UploadStreamVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd), log id: 627b6b6c >2014-10-23 15:15:11,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- executeVdsBrokerCommand: >2014-10-23 15:15:11,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGUID=afebca28-a0c7-47d5-9eea-6019e6dc038e > volUUID=28d85b18-cf5e-4af2-95c7-3075483fa61e > size=10,240 > >2014-10-23 15:15:11,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] FINISH, UploadStreamVDSCommand, log id: 627b6b6c >2014-10-23 15:15:11,772 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] CommandMultiAsyncTasks::AttachTask: Attaching task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 to command a252321e-f190-4295-a622-61466953cfc7. >2014-10-23 15:15:11,800 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] Adding task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:11,808 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS_EXECUTION >] >2014-10-23 15:15:11,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] START, SetVolumeDescriptionVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = afebca28-a0c7-47d5-9eea-6019e6dc038e, imageId = 28d85b18-cf5e-4af2-95c7-3075483fa61e), log id: 6d125394 >2014-10-23 15:15:11,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:11,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- setVolumeDescription parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGroupGUID=afebca28-a0c7-47d5-9eea-6019e6dc038e > volUUID=28d85b18-cf5e-4af2-95c7-3075483fa61e > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"6de9807c-ffcd-4be1-a76e-b5eaefbb7627"}],"Last Updated":"Thu Oct 23 15:15:11 IDT 2014","Size":10240} > >2014-10-23 15:15:11,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] FINISH, SetVolumeDescriptionVDSCommand, log id: 6d125394 >2014-10-23 15:15:11,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] START, SetVolumeDescriptionVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = 62a19a8f-7edd-45b6-9aaa-d585b545509f, imageId = 887d3915-35d1-48b1-bdd4-5f25751e050d), log id: 23b9502a >2014-10-23 15:15:11,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:11,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] -- setVolumeDescription parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGroupGUID=62a19a8f-7edd-45b6-9aaa-d585b545509f > volUUID=887d3915-35d1-48b1-bdd4-5f25751e050d > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"6de9807c-ffcd-4be1-a76e-b5eaefbb7627"}],"Last Updated":null} > >2014-10-23 15:15:11,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [2a1cef4b] FINISH, SetVolumeDescriptionVDSCommand, log id: 23b9502a >2014-10-23 15:15:11,989 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS_EXECUTION >] >2014-10-23 15:15:12,001 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] Running command: UploadStreamCommand internal: true. Entities affected : ID: 6de9807c-ffcd-4be1-a76e-b5eaefbb7627 Type: Storage >2014-10-23 15:15:12,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, UploadStreamVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd), log id: 3d226e1d >2014-10-23 15:15:12,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] -- executeVdsBrokerCommand: >2014-10-23 15:15:12,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] -- parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGUID=62a19a8f-7edd-45b6-9aaa-d585b545509f > volUUID=887d3915-35d1-48b1-bdd4-5f25751e050d > size=10,240 > >2014-10-23 15:15:12,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, UploadStreamVDSCommand, log id: 3d226e1d >2014-10-23 15:15:12,767 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] CommandMultiAsyncTasks::AttachTask: Attaching task 8b500e09-b424-4253-a392-92efbb4257a7 to command a252321e-f190-4295-a622-61466953cfc7. >2014-10-23 15:15:12,822 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] Adding task 8b500e09-b424-4253-a392-92efbb4257a7 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:12,826 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 38db43ff-c477-42e3-8fa2-780b6bf164cd value: VDS_EXECUTION >] >2014-10-23 15:15:12,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, SetVolumeDescriptionVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, storageDomainId = 6de9807c-ffcd-4be1-a76e-b5eaefbb7627, imageGroupId = 62a19a8f-7edd-45b6-9aaa-d585b545509f, imageId = 887d3915-35d1-48b1-bdd4-5f25751e050d), log id: 7f8d50d2 >2014-10-23 15:15:12,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:12,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] -- setVolumeDescription parameters: > spUUID=d4869232-1f76-44ed-9c03-2755258c43d5 > sdUUID=6de9807c-ffcd-4be1-a76e-b5eaefbb7627 > imageGroupGUID=62a19a8f-7edd-45b6-9aaa-d585b545509f > volUUID=887d3915-35d1-48b1-bdd4-5f25751e050d > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"6de9807c-ffcd-4be1-a76e-b5eaefbb7627"}],"Last Updated":"Thu Oct 23 15:15:11 IDT 2014","Size":10240} > >2014-10-23 15:15:12,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, SetVolumeDescriptionVDSCommand, log id: 7f8d50d2 >2014-10-23 15:15:12,907 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] BaseAsyncTask::startPollingTask: Starting to poll task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823. >2014-10-23 15:15:12,907 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] BaseAsyncTask::startPollingTask: Starting to poll task 8b500e09-b424-4253-a392-92efbb4257a7. >2014-10-23 15:15:12,907 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain completed, handling the result. >2014-10-23 15:15:12,907 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain succeeded, clearing tasks. >2014-10-23 15:15:12,907 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task bbdbef02-748b-480f-970e-ebfe06150fa2 >2014-10-23 15:15:12,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, SPMClearTaskVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, taskId = bbdbef02-748b-480f-970e-ebfe06150fa2), log id: 6e8569a6 >2014-10-23 15:15:12,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, taskId=bbdbef02-748b-480f-970e-ebfe06150fa2), log id: 78db49f9 >2014-10-23 15:15:12,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, HSMClearTaskVDSCommand, log id: 78db49f9 >2014-10-23 15:15:12,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, SPMClearTaskVDSCommand, log id: 6e8569a6 >2014-10-23 15:15:12,946 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] BaseAsyncTask::removeTaskFromDB: Removed task bbdbef02-748b-480f-970e-ebfe06150fa2 from DataBase >2014-10-23 15:15:12,946 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task b7f10dbd-a8ae-4e27-ab57-488370326327 >2014-10-23 15:15:12,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, SPMClearTaskVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, taskId = b7f10dbd-a8ae-4e27-ab57-488370326327), log id: 61027045 >2014-10-23 15:15:12,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, taskId=b7f10dbd-a8ae-4e27-ab57-488370326327), log id: 6718f3c4 >2014-10-23 15:15:12,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, HSMClearTaskVDSCommand, log id: 6718f3c4 >2014-10-23 15:15:12,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] FINISH, SPMClearTaskVDSCommand, log id: 61027045 >2014-10-23 15:15:13,004 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] BaseAsyncTask::removeTaskFromDB: Removed task b7f10dbd-a8ae-4e27-ab57-488370326327 from DataBase >2014-10-23 15:15:13,005 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-31) [7d532c4c] [within thread]: Some of the tasks related to command id a252321e-f190-4295-a622-61466953cfc7 were not cleared yet (Task id 8b500e09-b424-4253-a392-92efbb4257a7 is in state Polling). >2014-10-23 15:15:21,084 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) Polling and updating Async Tasks: 6 tasks, 4 tasks to poll now >2014-10-23 15:15:21,193 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) SPMAsyncTask::PollTask: Polling task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:21,193 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) BaseAsyncTask::onTaskEndSuccess: Task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:21,193 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-46) Task with DB Task ID 1183cb39-6212-4ff3-a2ad-19e69b17e28e and VDSM Task ID 8b500e09-b424-4253-a392-92efbb4257a7 is in state Polling. End action for command a252321e-f190-4295-a622-61466953cfc7 will proceed when all the entitys tasks are completed. >2014-10-23 15:15:21,193 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) SPMAsyncTask::PollTask: Polling task 8b500e09-b424-4253-a392-92efbb4257a7 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:21,194 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) BaseAsyncTask::onTaskEndSuccess: Task 8b500e09-b424-4253-a392-92efbb4257a7 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:21,194 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) CommandAsyncTask::endActionIfNecessary: All tasks of command a252321e-f190-4295-a622-61466953cfc7 has ended -> executing endAction >2014-10-23 15:15:21,194 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: a252321e-f190-4295-a622-61466953cfc7): calling endAction . >2014-10-23 15:15:21,194 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) Finished polling Tasks, will poll again in 10 seconds. >2014-10-23 15:15:21,196 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-23) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-10-23 15:15:21,221 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-10-23 15:15:21,222 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-10-23 15:15:21,222 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-10-23 15:15:21,222 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 8b500e09-b424-4253-a392-92efbb4257a7 >2014-10-23 15:15:21,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] START, SPMClearTaskVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, taskId = 8b500e09-b424-4253-a392-92efbb4257a7), log id: 780a2d86 >2014-10-23 15:15:21,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, taskId=8b500e09-b424-4253-a392-92efbb4257a7), log id: 6432f5fb >2014-10-23 15:15:21,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] FINISH, HSMClearTaskVDSCommand, log id: 6432f5fb >2014-10-23 15:15:21,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] FINISH, SPMClearTaskVDSCommand, log id: 780a2d86 >2014-10-23 15:15:21,317 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] BaseAsyncTask::removeTaskFromDB: Removed task 8b500e09-b424-4253-a392-92efbb4257a7 from DataBase >2014-10-23 15:15:21,317 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 >2014-10-23 15:15:21,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] START, SPMClearTaskVDSCommand( storagePoolId = d4869232-1f76-44ed-9c03-2755258c43d5, ignoreFailoverLimit = false, taskId = 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823), log id: 7e9c3a83 >2014-10-23 15:15:21,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] START, HSMClearTaskVDSCommand(HostName = monique-vds01.tlv.redhat.com, HostId = 38db43ff-c477-42e3-8fa2-780b6bf164cd, taskId=2abfa390-aa49-4bd6-a2eb-fc4fcbe67823), log id: 42c5af65 >2014-10-23 15:15:21,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] FINISH, HSMClearTaskVDSCommand, log id: 42c5af65 >2014-10-23 15:15:21,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-23) [13b733dc] FINISH, SPMClearTaskVDSCommand, log id: 7e9c3a83 >2014-10-23 15:15:21,405 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] BaseAsyncTask::removeTaskFromDB: Removed task 2abfa390-aa49-4bd6-a2eb-fc4fcbe67823 from DataBase >2014-10-23 15:15:21,405 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-23) [13b733dc] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity a252321e-f190-4295-a622-61466953cfc7 >2014-10-23 15:15:31,202 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) SPMAsyncTask::PollTask: Polling task 73304f9f-fa11-40bf-8047-64f15945c5c7 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:31,202 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) BaseAsyncTask::onTaskEndSuccess: Task 73304f9f-fa11-40bf-8047-64f15945c5c7 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:31,202 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-23) Task with DB Task ID 91a8e2a2-c92f-4edf-85b0-6c8f117a64ec and VDSM Task ID 28b4b9f8-25e8-435b-9bad-5d3450f26ddb is in state Polling. End action for command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 will proceed when all the entitys tasks are completed. >2014-10-23 15:15:31,203 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) SPMAsyncTask::PollTask: Polling task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:31,203 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-23) BaseAsyncTask::onTaskEndSuccess: Task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:31,203 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::endActionIfNecessary: All tasks of command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 has ended -> executing endAction >2014-10-23 15:15:31,203 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-23) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3): calling endAction . >2014-10-23 15:15:31,214 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-46) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction CreateOvfStoresForStorageDomain, executionIndex: 0 >2014-10-23 15:15:31,220 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] Ending command successfully: org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand >2014-10-23 15:15:31,252 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-46) [860ebd3] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 15:15:31,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-46) [860ebd3] START, GetImageInfoVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = dbc1563b-a64c-480e-b70b-aa8f76f95a5c, imageId = 5aa99f91-57c0-4ded-a828-4ec7a49a8c87), log id: 7a399e86 >2014-10-23 15:15:31,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-46) [860ebd3] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@49b89698, log id: 7a399e86 >2014-10-23 15:15:31,319 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-7-thread-46) [27d46590] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-10-23 15:15:31,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-46) [27d46590] START, GetImageInfoVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = 8a376036-bcdf-4cb2-a32b-689200d9b9d1, imageId = d7a67def-9bd7-431d-9ab3-ce9631f5cd22), log id: 586180d9 >2014-10-23 15:15:31,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-7-thread-46) [27d46590] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@64f4b34d, log id: 586180d9 >2014-10-23 15:15:31,366 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:31,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] START, SetVolumeDescriptionVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = 8a376036-bcdf-4cb2-a32b-689200d9b9d1, imageId = d7a67def-9bd7-431d-9ab3-ce9631f5cd22), log id: 322714c4 >2014-10-23 15:15:31,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:31,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] -- setVolumeDescription parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGroupGUID=8a376036-bcdf-4cb2-a32b-689200d9b9d1 > volUUID=d7a67def-9bd7-431d-9ab3-ce9631f5cd22 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc"}],"Last Updated":null} > >2014-10-23 15:15:31,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [6d020e16] FINISH, SetVolumeDescriptionVDSCommand, log id: 322714c4 >2014-10-23 15:15:31,467 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 15:15:31,476 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] Running command: UploadStreamCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:31,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] START, UploadStreamVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: abefae0 >2014-10-23 15:15:31,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- executeVdsBrokerCommand: >2014-10-23 15:15:31,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGUID=8a376036-bcdf-4cb2-a32b-689200d9b9d1 > volUUID=d7a67def-9bd7-431d-9ab3-ce9631f5cd22 > size=10,240 > >2014-10-23 15:15:31,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] FINISH, UploadStreamVDSCommand, log id: abefae0 >2014-10-23 15:15:31,729 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] CommandMultiAsyncTasks::AttachTask: Attaching task 47283459-867b-46d2-953c-06d33816b646 to command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3. >2014-10-23 15:15:31,753 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] Adding task 47283459-867b-46d2-953c-06d33816b646 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:31,761 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 15:15:31,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] START, SetVolumeDescriptionVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = 8a376036-bcdf-4cb2-a32b-689200d9b9d1, imageId = d7a67def-9bd7-431d-9ab3-ce9631f5cd22), log id: 57ae2a1d >2014-10-23 15:15:31,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:31,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- setVolumeDescription parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGroupGUID=8a376036-bcdf-4cb2-a32b-689200d9b9d1 > volUUID=d7a67def-9bd7-431d-9ab3-ce9631f5cd22 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc"}],"Last Updated":"Thu Oct 23 15:15:31 IDT 2014","Size":10240} > >2014-10-23 15:15:31,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] FINISH, SetVolumeDescriptionVDSCommand, log id: 57ae2a1d >2014-10-23 15:15:31,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] START, SetVolumeDescriptionVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = dbc1563b-a64c-480e-b70b-aa8f76f95a5c, imageId = 5aa99f91-57c0-4ded-a828-4ec7a49a8c87), log id: d443474 >2014-10-23 15:15:31,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:31,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] -- setVolumeDescription parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGroupGUID=dbc1563b-a64c-480e-b70b-aa8f76f95a5c > volUUID=5aa99f91-57c0-4ded-a828-4ec7a49a8c87 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc"}],"Last Updated":null} > >2014-10-23 15:15:31,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2fe17e41] FINISH, SetVolumeDescriptionVDSCommand, log id: d443474 >2014-10-23 15:15:31,961 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 15:15:31,966 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] Running command: UploadStreamCommand internal: true. Entities affected : ID: b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc Type: Storage >2014-10-23 15:15:31,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, UploadStreamVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44), log id: 3fe85dc6 >2014-10-23 15:15:31,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] -- executeVdsBrokerCommand: >2014-10-23 15:15:31,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] -- parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGUID=dbc1563b-a64c-480e-b70b-aa8f76f95a5c > volUUID=5aa99f91-57c0-4ded-a828-4ec7a49a8c87 > size=10,240 > >2014-10-23 15:15:32,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, UploadStreamVDSCommand, log id: 3fe85dc6 >2014-10-23 15:15:32,192 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] CommandMultiAsyncTasks::AttachTask: Attaching task 819988d4-df77-4ab1-97db-5524dd0e7d54 to command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3. >2014-10-23 15:15:32,205 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] Adding task 819988d4-df77-4ab1-97db-5524dd0e7d54 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-10-23 15:15:32,212 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: acb7e296-b82a-4376-9193-f26aedc8ef44 value: VDS_EXECUTION >] >2014-10-23 15:15:32,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, SetVolumeDescriptionVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, storageDomainId = b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc, imageGroupId = dbc1563b-a64c-480e-b70b-aa8f76f95a5c, imageId = 5aa99f91-57c0-4ded-a828-4ec7a49a8c87), log id: 4f5e7511 >2014-10-23 15:15:32,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-10-23 15:15:32,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] -- setVolumeDescription parameters: > spUUID=6705343f-9285-4942-be95-21ea33bb0125 > sdUUID=b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc > imageGroupGUID=dbc1563b-a64c-480e-b70b-aa8f76f95a5c > volUUID=5aa99f91-57c0-4ded-a828-4ec7a49a8c87 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"b6de6fc7-ce55-4fc2-ae7f-d7c9d629fcfc"}],"Last Updated":"Thu Oct 23 15:15:31 IDT 2014","Size":10240} > >2014-10-23 15:15:32,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, SetVolumeDescriptionVDSCommand, log id: 4f5e7511 >2014-10-23 15:15:32,276 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] BaseAsyncTask::startPollingTask: Starting to poll task 47283459-867b-46d2-953c-06d33816b646. >2014-10-23 15:15:32,277 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] BaseAsyncTask::startPollingTask: Starting to poll task 819988d4-df77-4ab1-97db-5524dd0e7d54. >2014-10-23 15:15:32,277 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain completed, handling the result. >2014-10-23 15:15:32,277 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain succeeded, clearing tasks. >2014-10-23 15:15:32,277 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb >2014-10-23 15:15:32,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, SPMClearTaskVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, taskId = 28b4b9f8-25e8-435b-9bad-5d3450f26ddb), log id: 950fea4 >2014-10-23 15:15:32,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=28b4b9f8-25e8-435b-9bad-5d3450f26ddb), log id: 247dc5c3 >2014-10-23 15:15:32,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, HSMClearTaskVDSCommand, log id: 247dc5c3 >2014-10-23 15:15:32,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, SPMClearTaskVDSCommand, log id: 950fea4 >2014-10-23 15:15:32,335 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] BaseAsyncTask::removeTaskFromDB: Removed task 28b4b9f8-25e8-435b-9bad-5d3450f26ddb from DataBase >2014-10-23 15:15:32,335 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 73304f9f-fa11-40bf-8047-64f15945c5c7 >2014-10-23 15:15:32,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, SPMClearTaskVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, taskId = 73304f9f-fa11-40bf-8047-64f15945c5c7), log id: d98d15 >2014-10-23 15:15:32,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=73304f9f-fa11-40bf-8047-64f15945c5c7), log id: 4abd5e22 >2014-10-23 15:15:32,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, HSMClearTaskVDSCommand, log id: 4abd5e22 >2014-10-23 15:15:32,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] FINISH, SPMClearTaskVDSCommand, log id: d98d15 >2014-10-23 15:15:32,366 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] BaseAsyncTask::removeTaskFromDB: Removed task 73304f9f-fa11-40bf-8047-64f15945c5c7 from DataBase >2014-10-23 15:15:32,366 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-7-thread-46) [2e3cf468] [within thread]: Some of the tasks related to command id 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 were not cleared yet (Task id 819988d4-df77-4ab1-97db-5524dd0e7d54 is in state Polling). >2014-10-23 15:15:41,204 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-4) Polling and updating Async Tasks: 8 tasks, 2 tasks to poll now >2014-10-23 15:15:41,211 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-4) SPMAsyncTask::PollTask: Polling task 819988d4-df77-4ab1-97db-5524dd0e7d54 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:41,211 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-4) BaseAsyncTask::onTaskEndSuccess: Task 819988d4-df77-4ab1-97db-5524dd0e7d54 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:41,211 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-4) Task with DB Task ID 5d4e632a-57ea-4e28-ab88-a73c53d88579 and VDSM Task ID 47283459-867b-46d2-953c-06d33816b646 is in state Polling. End action for command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 will proceed when all the entitys tasks are completed. >2014-10-23 15:15:41,211 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-4) SPMAsyncTask::PollTask: Polling task 47283459-867b-46d2-953c-06d33816b646 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-10-23 15:15:41,212 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-4) BaseAsyncTask::onTaskEndSuccess: Task 47283459-867b-46d2-953c-06d33816b646 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-10-23 15:15:41,212 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-4) CommandAsyncTask::endActionIfNecessary: All tasks of command 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 has ended -> executing endAction >2014-10-23 15:15:41,212 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-4) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3): calling endAction . >2014-10-23 15:15:41,218 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-16) CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-10-23 15:15:41,235 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-10-23 15:15:41,235 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-10-23 15:15:41,236 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-10-23 15:15:41,236 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 819988d4-df77-4ab1-97db-5524dd0e7d54 >2014-10-23 15:15:41,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] START, SPMClearTaskVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, taskId = 819988d4-df77-4ab1-97db-5524dd0e7d54), log id: 6eb91be3 >2014-10-23 15:15:41,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=819988d4-df77-4ab1-97db-5524dd0e7d54), log id: 3ad60cdf >2014-10-23 15:15:41,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] FINISH, HSMClearTaskVDSCommand, log id: 3ad60cdf >2014-10-23 15:15:41,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] FINISH, SPMClearTaskVDSCommand, log id: 6eb91be3 >2014-10-23 15:15:41,260 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] BaseAsyncTask::removeTaskFromDB: Removed task 819988d4-df77-4ab1-97db-5524dd0e7d54 from DataBase >2014-10-23 15:15:41,260 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 47283459-867b-46d2-953c-06d33816b646 >2014-10-23 15:15:41,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] START, SPMClearTaskVDSCommand( storagePoolId = 6705343f-9285-4942-be95-21ea33bb0125, ignoreFailoverLimit = false, taskId = 47283459-867b-46d2-953c-06d33816b646), log id: 6dddc2a8 >2014-10-23 15:15:41,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] START, HSMClearTaskVDSCommand(HostName = monique-vds05.tlv.redhat.com, HostId = acb7e296-b82a-4376-9193-f26aedc8ef44, taskId=47283459-867b-46d2-953c-06d33816b646), log id: 7a16dbf1 >2014-10-23 15:15:41,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] FINISH, HSMClearTaskVDSCommand, log id: 7a16dbf1 >2014-10-23 15:15:41,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-7-thread-16) [6d020e16] FINISH, SPMClearTaskVDSCommand, log id: 6dddc2a8 >2014-10-23 15:15:41,286 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] BaseAsyncTask::removeTaskFromDB: Removed task 47283459-867b-46d2-953c-06d33816b646 from DataBase >2014-10-23 15:15:41,286 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-7-thread-16) [6d020e16] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 32d2ca2f-81cf-4dde-8ff9-f452a64cf2b3 >2014-10-23 15:16:26,551 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Setting new tasks map. The map contains now 4 tasks >2014-10-23 15:16:26,551 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Cleared all tasks of pool d4869232-1f76-44ed-9c03-2755258c43d5. >2014-10-23 15:16:56,552 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-76) Setting new tasks map. The map contains now 0 tasks >2014-10-23 15:16:56,552 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-76) Cleared all tasks of pool 6705343f-9285-4942-be95-21ea33bb0125. >2014-10-23 15:22:03,982 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:22:04,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:22:04,010 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [299d2cde] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:22:04,016 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [61cc2bfe] Running command: LogoutUserCommand internal: false. >2014-10-23 15:22:04,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [61cc2bfe] Correlation ID: 61cc2bfe, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:22:12,721 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:22:12,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:22:12,746 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [27a77551] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:22:12,749 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [ad07ca3] Running command: LogoutUserCommand internal: false. >2014-10-23 15:22:12,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [ad07ca3] Correlation ID: ad07ca3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:22:17,026 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:22:17,053 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [4d37248] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:22:17,056 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [476e7bc0] Running command: LogoutUserCommand internal: false. >2014-10-23 15:22:17,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [476e7bc0] Correlation ID: 476e7bc0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:22:26,916 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:22:26,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:22:26,938 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [975f58c] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:22:26,941 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [7de8c6cc] Running command: LogoutUserCommand internal: false. >2014-10-23 15:22:26,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [7de8c6cc] Correlation ID: 7de8c6cc, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:22:52,093 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:22:52,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:23:05,476 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:23:05,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:23:05,484 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [62d98c79] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:23:05,487 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [2c7d217d] Running command: LogoutUserCommand internal: false. >2014-10-23 15:23:05,490 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [2c7d217d] Correlation ID: 2c7d217d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:23:39,261 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:23:39,301 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:23:39,306 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [4f12a7ea] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:23:39,310 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [582bc847] Running command: LogoutUserCommand internal: false. >2014-10-23 15:23:39,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [582bc847] Correlation ID: 582bc847, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:23:52,159 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:23:52,161 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:24:31,870 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:24:31,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:24:31,878 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [539c8ffb] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:24:31,880 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [19ccadf4] Running command: LogoutUserCommand internal: false. >2014-10-23 15:24:31,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [19ccadf4] Correlation ID: 19ccadf4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:24:34,028 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:24:34,033 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [4c448159] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:24:34,035 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [79e006f1] Running command: LogoutUserCommand internal: false. >2014-10-23 15:24:34,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [79e006f1] Correlation ID: 79e006f1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:24:38,577 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:24:38,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:24:38,586 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [428cf534] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:24:38,588 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [5c173b24] Running command: LogoutUserCommand internal: false. >2014-10-23 15:24:38,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [5c173b24] Correlation ID: 5c173b24, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:24:52,155 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:24:52,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:26:05,025 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:26:05,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:26:05,037 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [29e87893] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:26:05,040 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [1ddefd88] Running command: LogoutUserCommand internal: false. >2014-10-23 15:26:05,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [1ddefd88] Correlation ID: 1ddefd88, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:26:52,188 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:26:52,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:30:03,513 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:30:03,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:30:03,549 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [2bb8524e] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:30:03,553 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [12c13fa9] Running command: LogoutUserCommand internal: false. >2014-10-23 15:30:03,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [12c13fa9] Correlation ID: 12c13fa9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:30:08,490 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:30:08,500 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [3448f063] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:30:08,514 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [5edf8d4f] Running command: LogoutUserCommand internal: false. >2014-10-23 15:30:08,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [5edf8d4f] Correlation ID: 5edf8d4f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:31:20,410 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:31:20,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:31:20,424 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [1be8f221] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:31:20,437 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [44cfbb70] Running command: LogoutUserCommand internal: false. >2014-10-23 15:31:20,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [44cfbb70] Correlation ID: 44cfbb70, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:31:32,536 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:31:32,570 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:31:32,576 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [7f21f07b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:31:32,579 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [32693e41] Running command: LogoutUserCommand internal: false. >2014-10-23 15:31:32,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [32693e41] Correlation ID: 32693e41, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:31:42,449 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:31:42,454 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:31:42,469 INFO [org.ovirt.engine.core.bll.AddExternalStepCommand] (ajp-/127.0.0.1:8702-4) [8e9698a] Running command: AddExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:31:43,573 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [5d8050de] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:31:43,587 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [72378f76] Running command: LogoutUserCommand internal: false. >2014-10-23 15:31:43,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [72378f76] Correlation ID: 72378f76, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:32:44,058 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:32:44,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:32:44,103 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [714fac63] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:32:44,105 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [21c052d] Running command: LogoutUserCommand internal: false. >2014-10-23 15:32:44,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [21c052d] Correlation ID: 21c052d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:32:55,988 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:32:55,991 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:32:56,006 INFO [org.ovirt.engine.core.bll.AddExternalStepCommand] (ajp-/127.0.0.1:8702-8) [7f549e7f] Running command: AddExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:32:56,027 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [2b942bd0] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:32:56,030 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [326f2df9] Running command: LogoutUserCommand internal: false. >2014-10-23 15:32:56,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [326f2df9] Correlation ID: 326f2df9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:33:00,566 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:33:00,582 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [727345a2] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:33:00,592 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [2f6efc2d] Running command: LogoutUserCommand internal: false. >2014-10-23 15:33:00,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [2f6efc2d] Correlation ID: 2f6efc2d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:33:36,297 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:33:36,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:33:36,316 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [57040a49] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:33:36,319 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [16a22357] Running command: LogoutUserCommand internal: false. >2014-10-23 15:33:36,325 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [16a22357] Correlation ID: 16a22357, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:33:52,384 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:33:52,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:33:53,931 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:33:53,946 INFO [org.ovirt.engine.core.bll.AddExternalStepCommand] (ajp-/127.0.0.1:8702-1) [53f2d40b] Running command: AddExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:33:53,960 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [7e4c77b5] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:33:53,963 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [15657843] Running command: LogoutUserCommand internal: false. >2014-10-23 15:33:53,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [15657843] Correlation ID: 15657843, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:34:17,433 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:34:17,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:34:17,454 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [5246168f] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:34:17,456 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [1a503644] Running command: LogoutUserCommand internal: false. >2014-10-23 15:34:17,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [1a503644] Correlation ID: 1a503644, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:35:01,068 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:35:01,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:35:01,081 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [2d235d65] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:35:01,107 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [66bd7159] Running command: LogoutUserCommand internal: false. >2014-10-23 15:35:01,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [66bd7159] Correlation ID: 66bd7159, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:35:08,698 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:35:08,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:35:08,779 INFO [org.ovirt.engine.core.bll.AddExternalStepCommand] (ajp-/127.0.0.1:8702-3) [5b2bf7ae] Running command: AddExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:35:08,792 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [22517514] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:35:08,794 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [66ad2fd7] Running command: LogoutUserCommand internal: false. >2014-10-23 15:35:08,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [66ad2fd7] Correlation ID: 66ad2fd7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:35:52,408 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:35:52,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:36:31,591 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:36:31,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:36:31,643 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [47425ebb] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:36:31,646 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [48d5aa7d] Running command: LogoutUserCommand internal: false. >2014-10-23 15:36:31,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [48d5aa7d] Correlation ID: 48d5aa7d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:36:52,429 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:36:52,455 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:37:54,096 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:37:54,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:37:54,160 INFO [org.ovirt.engine.core.bll.AddExternalStepCommand] (ajp-/127.0.0.1:8702-11) [2234fa3e] Running command: AddExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:37:54,199 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [781635c4] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:37:54,201 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [4f0a56c6] Running command: LogoutUserCommand internal: false. >2014-10-23 15:37:54,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [4f0a56c6] Correlation ID: 4f0a56c6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:38:36,345 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:38:36,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:38:36,358 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [61310cdb] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:38:36,370 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [7c005244] Running command: LogoutUserCommand internal: false. >2014-10-23 15:38:36,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [7c005244] Correlation ID: 7c005244, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:38:45,085 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:38:45,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:38:45,128 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [36e19967] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:38:45,130 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [5a9f9a28] Running command: LogoutUserCommand internal: false. >2014-10-23 15:38:45,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [5a9f9a28] Correlation ID: 5a9f9a28, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:38:52,461 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:38:52,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:38:54,266 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:38:54,294 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [34e6deeb] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:38:54,297 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [11868ec9] Running command: LogoutUserCommand internal: false. >2014-10-23 15:38:54,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [11868ec9] Correlation ID: 11868ec9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:38:54,395 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:38:54,416 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [6efbc1ef] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:38:54,432 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [7be8f82] Running command: LogoutUserCommand internal: false. >2014-10-23 15:38:54,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [7be8f82] Correlation ID: 7be8f82, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:39:52,479 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:39:52,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:11,164 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:11,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:11,197 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [63220f7c] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:42:11,200 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [4369bf4f] Running command: LogoutUserCommand internal: false. >2014-10-23 15:42:11,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [4369bf4f] Correlation ID: 4369bf4f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:42:20,128 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:20,137 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:20,140 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [7f785547] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:42:20,155 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [273782db] Running command: LogoutUserCommand internal: false. >2014-10-23 15:42:20,161 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [273782db] Correlation ID: 273782db, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:42:27,624 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:27,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:27,630 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [1750ec8b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:42:27,633 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [714c7d8c] Running command: LogoutUserCommand internal: false. >2014-10-23 15:42:27,669 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [714c7d8c] Correlation ID: 714c7d8c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:42:36,794 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:36,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:36,801 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [31bbeef8] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:42:36,806 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [6d3e22eb] Running command: LogoutUserCommand internal: false. >2014-10-23 15:42:36,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [6d3e22eb] Correlation ID: 6d3e22eb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:42:52,531 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:52,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:42:56,961 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:42:56,979 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [5d6260ef] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:42:56,981 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [5fd60641] Running command: LogoutUserCommand internal: false. >2014-10-23 15:42:56,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [5fd60641] Correlation ID: 5fd60641, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:43:32,734 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:43:32,764 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:43:33,754 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-3) [185c51e5] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:43:33,758 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [3bf081d9] Running command: LogoutUserCommand internal: false. >2014-10-23 15:43:33,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [3bf081d9] Correlation ID: 3bf081d9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:43:52,629 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:43:52,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:43:55,504 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:44:18,829 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:44:18,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:44:18,853 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [3ca06320] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:44:18,872 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [50c98991] Running command: LogoutUserCommand internal: false. >2014-10-23 15:44:18,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [50c98991] Correlation ID: 50c98991, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:44:32,681 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:44:32,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:44:32,717 INFO [org.ovirt.engine.core.bll.EndExternalJobCommand] (ajp-/127.0.0.1:8702-1) [7e3f0191] Running command: EndExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:44:33,814 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [2905092f] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:44:33,834 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [31368730] Running command: LogoutUserCommand internal: false. >2014-10-23 15:44:33,858 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [31368730] Correlation ID: 31368730, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:44:52,559 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:44:52,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:45:04,644 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:45:04,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:45:04,665 INFO [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-1) [f644761] Running command: ClearExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:45:04,674 ERROR [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-1) [f644761] Transaction rolled-back for command: org.ovirt.engine.core.bll.ClearExternalJobCommand. >2014-10-23 15:45:04,676 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-1) Operation Failed: [] >2014-10-23 15:45:04,701 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [34b8e7d] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:45:04,703 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [67623ec6] Running command: LogoutUserCommand internal: false. >2014-10-23 15:45:04,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [67623ec6] Correlation ID: 67623ec6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:45:52,822 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:45:52,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:45:55,541 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:45:55,555 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [3185c9cd] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:45:55,568 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [402766eb] Running command: LogoutUserCommand internal: false. >2014-10-23 15:45:55,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [402766eb] Correlation ID: 402766eb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:46:04,449 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:46:04,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:46:04,472 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [66474ee2] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:46:04,474 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [3525b34f] Running command: LogoutUserCommand internal: false. >2014-10-23 15:46:04,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [3525b34f] Correlation ID: 3525b34f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:46:16,895 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:46:16,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:46:16,906 INFO [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-5) [74ae721d] Running command: ClearExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:46:16,911 ERROR [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-5) [74ae721d] Transaction rolled-back for command: org.ovirt.engine.core.bll.ClearExternalJobCommand. >2014-10-23 15:46:16,912 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-5) Operation Failed: [] >2014-10-23 15:46:16,916 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [381e37b6] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:46:16,919 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [7997cffa] Running command: LogoutUserCommand internal: false. >2014-10-23 15:46:16,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [7997cffa] Correlation ID: 7997cffa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:46:22,402 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:46:22,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:46:22,411 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [431c58de] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:46:22,422 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [4cb45d7a] Running command: LogoutUserCommand internal: false. >2014-10-23 15:46:22,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [4cb45d7a] Correlation ID: 4cb45d7a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:46:22,521 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:46:22,527 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [6018f104] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:46:22,529 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [3c54eae3] Running command: LogoutUserCommand internal: false. >2014-10-23 15:46:22,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [3c54eae3] Correlation ID: 3c54eae3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:46:52,599 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:46:52,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:49:01,246 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:01,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:49:01,292 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [18dc91ca] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:49:01,294 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [691679a1] Running command: LogoutUserCommand internal: false. >2014-10-23 15:49:01,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [691679a1] Correlation ID: 691679a1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:49:07,525 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:07,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:49:07,584 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [f48f25f] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:49:07,587 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [5290476c] Running command: LogoutUserCommand internal: false. >2014-10-23 15:49:07,590 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [5290476c] Correlation ID: 5290476c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:49:34,484 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:34,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:49:34,519 INFO [org.ovirt.engine.core.bll.EndExternalStepCommand] (ajp-/127.0.0.1:8702-1) [3ff7a070] Running command: EndExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:49:34,548 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [9f3373b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:49:34,551 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [447828f9] Running command: LogoutUserCommand internal: false. >2014-10-23 15:49:34,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [447828f9] Correlation ID: 447828f9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:49:52,608 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:52,614 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:49:53,892 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:53,914 INFO [org.ovirt.engine.core.bll.AddExternalJobCommand] (ajp-/127.0.0.1:8702-7) [1646cb30] Running command: AddExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:49:53,935 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [1646cb30] Correlation ID: 1646cb30, Call Stack: null, Custom Event ID: -1, Message: New external Job parent POM for all oVirt modules was added by user admin >2014-10-23 15:49:53,946 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [21512ecc] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:49:53,948 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [5801e4ab] Running command: LogoutUserCommand internal: false. >2014-10-23 15:49:53,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [5801e4ab] Correlation ID: 5801e4ab, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:49:54,007 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:49:54,014 INFO [org.ovirt.engine.core.bll.AddExternalJobCommand] (ajp-/127.0.0.1:8702-5) [423fdf7a] Running command: AddExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:49:54,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [423fdf7a] Correlation ID: 423fdf7a, Call Stack: null, Custom Event ID: -1, Message: New external Job parent POM for all oVirt modules was added by user admin >2014-10-23 15:49:54,042 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [7ad3520a] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:49:54,045 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [35c8019b] Running command: LogoutUserCommand internal: false. >2014-10-23 15:49:54,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [35c8019b] Correlation ID: 35c8019b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:01,415 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-10-23 15:50:01,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:50:01,423 INFO [org.ovirt.engine.core.bll.AddExternalJobCommand] (ajp-/127.0.0.1:8702-10) [2d113589] Running command: AddExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:50:01,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [2d113589] Correlation ID: 2d113589, Call Stack: null, Custom Event ID: -1, Message: New external Job parent POM for all oVirt modules was added by user admin >2014-10-23 15:50:01,448 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-10) [117b4a0c] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:50:01,451 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [5e27abeb] Running command: LogoutUserCommand internal: false. >2014-10-23 15:50:01,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [5e27abeb] Correlation ID: 5e27abeb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:10,801 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:50:10,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:50:10,868 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [2387f224] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:50:10,871 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [776430af] Running command: LogoutUserCommand internal: false. >2014-10-23 15:50:10,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [776430af] Correlation ID: 776430af, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:27,869 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:50:27,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:50:27,888 INFO [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-4) [7bb95f0d] Running command: ClearExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:50:27,891 ERROR [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-4) [7bb95f0d] Transaction rolled-back for command: org.ovirt.engine.core.bll.ClearExternalJobCommand. >2014-10-23 15:50:27,891 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-4) Operation Failed: [] >2014-10-23 15:50:27,906 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [75fd2bbd] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:50:27,910 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [4a1dbb14] Running command: LogoutUserCommand internal: false. >2014-10-23 15:50:27,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [4a1dbb14] Correlation ID: 4a1dbb14, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:36,657 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:50:36,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:50:36,681 INFO [org.ovirt.engine.core.bll.EndExternalStepCommand] (ajp-/127.0.0.1:8702-1) [19cbfbc8] Running command: EndExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:50:36,748 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [6f5afafd] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:50:36,751 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [462a350e] Running command: LogoutUserCommand internal: false. >2014-10-23 15:50:36,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [462a350e] Correlation ID: 462a350e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:52,234 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:50:52,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:50:52,279 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [5b8d3ff] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:50:52,281 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [77ca3ab2] Running command: LogoutUserCommand internal: false. >2014-10-23 15:50:52,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [77ca3ab2] Correlation ID: 77ca3ab2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:50:52,646 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:03,297 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:03,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:03,307 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [3ab0d567] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:03,310 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [33590750] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:03,336 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [33590750] Correlation ID: 33590750, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:09,006 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:09,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:09,027 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:09,031 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [548cb53c] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:09,033 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [596676f8] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:09,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [596676f8] Correlation ID: 596676f8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:09,048 INFO [org.ovirt.engine.core.bll.EndExternalStepCommand] (ajp-/127.0.0.1:8702-4) [6b7ea098] Running command: EndExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:51:09,058 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [75f91b7f] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:09,083 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [407771d4] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:09,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [407771d4] Correlation ID: 407771d4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:21,680 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:21,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:21,762 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [ba968c4] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:21,765 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [736d8fea] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:21,773 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [736d8fea] Correlation ID: 736d8fea, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:29,555 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:29,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:29,591 INFO [org.ovirt.engine.core.bll.EndExternalStepCommand] (ajp-/127.0.0.1:8702-11) [5478a3e4] Running command: EndExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:51:29,599 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [2450ec57] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:29,601 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [30c80a2c] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:29,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [30c80a2c] Correlation ID: 30c80a2c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:31,975 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:31,997 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [19fe2eef] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:32,000 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [726436ca] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:32,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [726436ca] Correlation ID: 726436ca, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:36,966 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:36,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:36,991 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [5105078e] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:36,994 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [2404e4c7] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:37,018 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [2404e4c7] Correlation ID: 2404e4c7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:41,880 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:41,899 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [5d992337] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:41,978 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [6938767d] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:42,000 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [6938767d] Correlation ID: 6938767d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:43,167 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:43,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:43,194 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [5e821a35] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:43,197 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [2dcab67f] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:43,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [2dcab67f] Correlation ID: 2dcab67f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:51,700 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-10-23 15:51:51,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:51:51,718 INFO [org.ovirt.engine.core.bll.EndExternalStepCommand] (ajp-/127.0.0.1:8702-2) [24f769b4] Running command: EndExternalStepCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:51:51,726 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-2) [467ee753] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:51:51,729 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [59533a80] Running command: LogoutUserCommand internal: false. >2014-10-23 15:51:51,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [59533a80] Correlation ID: 59533a80, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:51:52,684 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-10-23 15:52:13,141 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:52:13,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:52:13,166 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [1ed02dff] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:52:13,169 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [5a9859d4] Running command: LogoutUserCommand internal: false. >2014-10-23 15:52:13,199 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [5a9859d4] Correlation ID: 5a9859d4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:52:28,274 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-10-23 15:52:28,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:52:28,342 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 15:52:28,389 INFO [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-6) [42edb951] Running command: ClearExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:52:28,393 ERROR [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-6) [42edb951] Transaction rolled-back for command: org.ovirt.engine.core.bll.ClearExternalJobCommand. >2014-10-23 15:52:28,393 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-6) Operation Failed: [] >2014-10-23 15:52:28,396 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [603af952] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:52:28,398 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [372eca91] Running command: LogoutUserCommand internal: false. >2014-10-23 15:52:28,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [372eca91] Correlation ID: 372eca91, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:52:52,639 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:52:52,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:53:08,695 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:53:08,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:53:08,710 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [77556f1d] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:53:08,713 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [72e6062f] Running command: LogoutUserCommand internal: false. >2014-10-23 15:53:08,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [72e6062f] Correlation ID: 72e6062f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:53:52,664 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:53:52,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:56:41,395 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-10-23 15:56:41,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:56:41,416 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-7) [124d5b7a] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:56:41,419 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [4146ac] Running command: LogoutUserCommand internal: false. >2014-10-23 15:56:41,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [4146ac] Correlation ID: 4146ac, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:56:57,356 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 15:56:57,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:56:57,396 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-1) [4ceb4a24] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:56:57,399 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [37f8e4e1] Running command: LogoutUserCommand internal: false. >2014-10-23 15:56:57,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [37f8e4e1] Correlation ID: 37f8e4e1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:57:01,573 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-10-23 15:57:01,593 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [7807831] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:57:01,596 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [165ff05e] Running command: LogoutUserCommand internal: false. >2014-10-23 15:57:01,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [165ff05e] Correlation ID: 165ff05e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:58:05,422 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 15:58:05,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:58:05,498 INFO [org.ovirt.engine.core.bll.EndExternalJobCommand] (ajp-/127.0.0.1:8702-5) [23ad45c9] Running command: EndExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:58:05,510 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [6ff074e8] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:58:05,512 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [1187758b] Running command: LogoutUserCommand internal: false. >2014-10-23 15:58:05,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [1187758b] Correlation ID: 1187758b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:58:45,381 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-10-23 15:58:45,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:58:45,423 INFO [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-8) [4e4adc87] Running command: ClearExternalJobCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group INJECT_EXTERNAL_TASKS with role type ADMIN >2014-10-23 15:58:45,428 ERROR [org.ovirt.engine.core.bll.ClearExternalJobCommand] (ajp-/127.0.0.1:8702-8) [4e4adc87] Transaction rolled-back for command: org.ovirt.engine.core.bll.ClearExternalJobCommand. >2014-10-23 15:58:45,428 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-8) Operation Failed: [] >2014-10-23 15:58:45,432 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-8) [73e20047] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:58:45,435 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [53ae3e11] Running command: LogoutUserCommand internal: false. >2014-10-23 15:58:45,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [53ae3e11] Correlation ID: 53ae3e11, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 15:59:59,763 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-10-23 15:59:59,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 15:59:59,799 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [c743d4b] Running command: LogoutBySessionCommand internal: false. >2014-10-23 15:59:59,801 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [71204e97] Running command: LogoutUserCommand internal: false. >2014-10-23 15:59:59,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [71204e97] Correlation ID: 71204e97, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 16:14:58,753 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-10-23 16:15:05,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-10-23 16:33:00,915 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-10-23 16:33:00,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 16:33:00,946 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-5) [18d65bc2] Running command: LogoutBySessionCommand internal: false. >2014-10-23 16:33:00,949 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [33b82814] Running command: LogoutUserCommand internal: false. >2014-10-23 16:33:00,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [33b82814] Correlation ID: 33b82814, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 16:33:08,558 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2014-10-23 16:33:08,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-10-23 16:33:08,579 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-6) [4834473e] Running command: LogoutBySessionCommand internal: false. >2014-10-23 16:33:08,581 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [473c1645] Running command: LogoutUserCommand internal: false. >2014-10-23 16:33:08,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [473c1645] Correlation ID: 473c1645, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-23 16:33:53,319 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-10-23 16:33:53,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1156055
: 949888