Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 664728 Details for
Bug 887751
Webadmin | MTU setting for network under cluster should be gray out for cluster 3.1 under DC 3.0
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine.log
engine.mtu_30.log (text/x-log), 4.30 MB, created by
Meni Yakove
on 2012-12-17 08:58:38 UTC
(
hide
)
Description:
engine.log
Filename:
MIME Type:
Creator:
Meni Yakove
Created:
2012-12-17 08:58:38 UTC
Size:
4.30 MB
patch
obsolete
>2012-12-17 08:16:28,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] FINISH, ResetIrsVDSCommand, log id: 5bc76910 >2012-12-17 08:16:28,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34c09940 >2012-12-17 08:16:28,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:28,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [7b78f9c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34c09940 >2012-12-17 08:16:28,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:16:28,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4489 >2012-12-17 08:16:28,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4489, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4225de10 >2012-12-17 08:16:28,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling started: taskId = 52fec17c-28ea-4404-b5d6-547238e0502c >2012-12-17 08:16:29,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling ended: taskId = 52fec17c-28ea-4404-b5d6-547238e0502c task status = finished >2012-12-17 08:16:29,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] spmStart polling ended, spm status: SPM >2012-12-17 08:16:29,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=52fec17c-28ea-4404-b5d6-547238e0502c), log id: 591a05be >2012-12-17 08:16:29,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMClearTaskVDSCommand, log id: 591a05be >2012-12-17 08:16:29,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b6f2aa5, log id: 4225de10 >2012-12-17 08:16:29,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:16:29,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:29,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4f9e16d3 >2012-12-17 08:16:29,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4f9e16d3 >2012-12-17 08:16:29,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2feb4141 >2012-12-17 08:16:29,732 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [3cf5d2fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:16:29,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13ec9f7b >2012-12-17 08:16:29,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:29,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1073bebc >2012-12-17 08:16:29,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1073bebc >2012-12-17 08:16:29,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13ec9f7b >2012-12-17 08:16:29,753 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [3cf5d2fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:16:34,177 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:16:38,660 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5ee35c25] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:16:38,799 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [f4188d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:16:38,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4358f48e >2012-12-17 08:16:38,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fd30bb7 >2012-12-17 08:16:38,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:39,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, SpmStopVDSCommand, log id: 2fd30bb7 >2012-12-17 08:16:39,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, ResetIrsVDSCommand, log id: 4358f48e >2012-12-17 08:16:39,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: dd5fbae >2012-12-17 08:16:39,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [f4188d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:39,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [f4188d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: dd5fbae >2012-12-17 08:16:44,346 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [58c4b5b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:16:49,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:16:49,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4490 >2012-12-17 08:16:49,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4490, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fb80809 >2012-12-17 08:16:49,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling started: taskId = b1a15a37-79f1-4ac5-b237-8e13829268ee >2012-12-17 08:16:50,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling ended: taskId = b1a15a37-79f1-4ac5-b237-8e13829268ee task status = finished >2012-12-17 08:16:50,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] spmStart polling ended, spm status: SPM >2012-12-17 08:16:50,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [740958cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1a15a37-79f1-4ac5-b237-8e13829268ee), log id: 49e9614e >2012-12-17 08:16:50,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [740958cc] FINISH, HSMClearTaskVDSCommand, log id: 49e9614e >2012-12-17 08:16:50,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [740958cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78f36dca, log id: 1fb80809 >2012-12-17 08:16:50,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:16:51,001 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:16:51,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 66f28248 >2012-12-17 08:16:51,053 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [740958cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:16:51,383 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [42c50c6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:16:51,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d18de24 >2012-12-17 08:16:51,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13cf339e >2012-12-17 08:16:51,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:51,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, SpmStopVDSCommand, log id: 13cf339e >2012-12-17 08:16:51,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, ResetIrsVDSCommand, log id: 1d18de24 >2012-12-17 08:16:51,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b33bc >2012-12-17 08:16:51,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:51,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [42c50c6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b33bc >2012-12-17 08:16:51,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:16:51,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4491 >2012-12-17 08:16:51,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4491, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56ae90dd >2012-12-17 08:16:51,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling started: taskId = 0abaa22a-a784-4345-b3fe-41e6859318ac >2012-12-17 08:16:52,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended: taskId = 0abaa22a-a784-4345-b3fe-41e6859318ac task status = finished >2012-12-17 08:16:52,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended, spm status: SPM >2012-12-17 08:16:52,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0abaa22a-a784-4345-b3fe-41e6859318ac), log id: 76a203d9 >2012-12-17 08:16:52,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMClearTaskVDSCommand, log id: 76a203d9 >2012-12-17 08:16:52,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6aad868, log id: 56ae90dd >2012-12-17 08:16:52,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:16:52,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:52,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 701085c2 >2012-12-17 08:16:52,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 701085c2 >2012-12-17 08:16:52,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 66f28248 >2012-12-17 08:16:52,924 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:16:52,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cc57f94 >2012-12-17 08:16:52,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:16:52,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63b628a9 >2012-12-17 08:16:52,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63b628a9 >2012-12-17 08:16:52,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cc57f94 >2012-12-17 08:16:52,944 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:16:54,612 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [284eeca1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:17:01,810 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [412bf4c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:01,988 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [16233e3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:17:02,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 42717f3 >2012-12-17 08:17:02,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d8a6a03 >2012-12-17 08:17:02,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:02,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, SpmStopVDSCommand, log id: 2d8a6a03 >2012-12-17 08:17:02,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, ResetIrsVDSCommand, log id: 42717f3 >2012-12-17 08:17:02,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b97a385 >2012-12-17 08:17:02,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:02,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b97a385 >2012-12-17 08:17:04,839 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:17:12,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:17:12,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4492 >2012-12-17 08:17:12,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4492, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15994090 >2012-12-17 08:17:12,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling started: taskId = 04d01d20-76bd-4401-a5d9-d6ff76d6d6e2 >2012-12-17 08:17:13,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling ended: taskId = 04d01d20-76bd-4401-a5d9-d6ff76d6d6e2 task status = finished >2012-12-17 08:17:13,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] spmStart polling ended, spm status: SPM >2012-12-17 08:17:13,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=04d01d20-76bd-4401-a5d9-d6ff76d6d6e2), log id: 1ffcfe67 >2012-12-17 08:17:13,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] FINISH, HSMClearTaskVDSCommand, log id: 1ffcfe67 >2012-12-17 08:17:13,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [28cae87e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1aa22c99, log id: 15994090 >2012-12-17 08:17:13,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:17:13,609 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:17:13,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 390ef54a >2012-12-17 08:17:13,615 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [28cae87e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:13,864 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [48110f6e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:17:13,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24c814a4 >2012-12-17 08:17:13,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11c984e3 >2012-12-17 08:17:14,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:14,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, SpmStopVDSCommand, log id: 11c984e3 >2012-12-17 08:17:14,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, ResetIrsVDSCommand, log id: 24c814a4 >2012-12-17 08:17:14,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44eeeac >2012-12-17 08:17:14,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:14,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [48110f6e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44eeeac >2012-12-17 08:17:14,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:17:14,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4493 >2012-12-17 08:17:14,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4493, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46c64d52 >2012-12-17 08:17:14,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling started: taskId = ba137dd1-b926-4f3f-a17a-a9a55be53ec6 >2012-12-17 08:17:15,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended: taskId = ba137dd1-b926-4f3f-a17a-a9a55be53ec6 task status = finished >2012-12-17 08:17:15,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended, spm status: SPM >2012-12-17 08:17:15,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ba137dd1-b926-4f3f-a17a-a9a55be53ec6), log id: 31277d4a >2012-12-17 08:17:15,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMClearTaskVDSCommand, log id: 31277d4a >2012-12-17 08:17:15,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17929e7c, log id: 46c64d52 >2012-12-17 08:17:15,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:17:16,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:16,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2abbb816 >2012-12-17 08:17:16,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2abbb816 >2012-12-17 08:17:16,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 390ef54a >2012-12-17 08:17:16,125 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:17:16,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a32c245 >2012-12-17 08:17:16,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:16,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cf02e36 >2012-12-17 08:17:16,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cf02e36 >2012-12-17 08:17:16,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a32c245 >2012-12-17 08:17:16,145 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:17:24,396 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [3637a35] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:24,571 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [406132e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:17:24,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b2310c2 >2012-12-17 08:17:24,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 402ad8a5 >2012-12-17 08:17:24,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:25,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStopVDSCommand, log id: 402ad8a5 >2012-12-17 08:17:25,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, ResetIrsVDSCommand, log id: 5b2310c2 >2012-12-17 08:17:25,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48c978e0 >2012-12-17 08:17:25,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:25,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48c978e0 >2012-12-17 08:17:25,859 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:17:35,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:17:35,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4494 >2012-12-17 08:17:35,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4494, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: fd4449e >2012-12-17 08:17:35,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling started: taskId = 94e60bdc-baad-477e-8735-7d19a04d9e32 >2012-12-17 08:17:37,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling ended: taskId = 94e60bdc-baad-477e-8735-7d19a04d9e32 task status = finished >2012-12-17 08:17:37,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] spmStart polling ended, spm status: SPM >2012-12-17 08:17:37,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=94e60bdc-baad-477e-8735-7d19a04d9e32), log id: 20b8102f >2012-12-17 08:17:37,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] FINISH, HSMClearTaskVDSCommand, log id: 20b8102f >2012-12-17 08:17:37,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [380e9d7e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@142e8e71, log id: fd4449e >2012-12-17 08:17:37,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:17:37,077 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:17:37,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 357be1d1 >2012-12-17 08:17:37,114 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [380e9d7e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:37,203 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:17:37,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e09901 >2012-12-17 08:17:37,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5549e288 >2012-12-17 08:17:37,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:37,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStopVDSCommand, log id: 5549e288 >2012-12-17 08:17:37,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, ResetIrsVDSCommand, log id: 6e09901 >2012-12-17 08:17:37,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 508e5d12 >2012-12-17 08:17:37,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:37,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 508e5d12 >2012-12-17 08:17:37,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:17:37,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4495 >2012-12-17 08:17:37,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4495, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69687062 >2012-12-17 08:17:37,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling started: taskId = 2e548bc5-8d22-4e18-aeb6-f153fc3276aa >2012-12-17 08:17:38,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended: taskId = 2e548bc5-8d22-4e18-aeb6-f153fc3276aa task status = finished >2012-12-17 08:17:38,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended, spm status: SPM >2012-12-17 08:17:38,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e548bc5-8d22-4e18-aeb6-f153fc3276aa), log id: 5417c0fc >2012-12-17 08:17:38,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMClearTaskVDSCommand, log id: 5417c0fc >2012-12-17 08:17:38,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@637f0b85, log id: 69687062 >2012-12-17 08:17:38,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:17:38,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:38,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41233a21 >2012-12-17 08:17:38,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41233a21 >2012-12-17 08:17:38,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 357be1d1 >2012-12-17 08:17:38,692 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:17:38,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1777a538 >2012-12-17 08:17:38,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:38,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c9e319c >2012-12-17 08:17:38,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c9e319c >2012-12-17 08:17:38,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1777a538 >2012-12-17 08:17:38,713 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:17:46,342 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:17:47,574 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [54d258ba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:47,698 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [667bec9e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:17:47,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5435fe04 >2012-12-17 08:17:47,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b062d28 >2012-12-17 08:17:47,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:48,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, SpmStopVDSCommand, log id: 3b062d28 >2012-12-17 08:17:48,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, ResetIrsVDSCommand, log id: 5435fe04 >2012-12-17 08:17:48,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1126be8a >2012-12-17 08:17:48,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [667bec9e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:17:48,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [667bec9e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1126be8a >2012-12-17 08:17:56,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:17:58,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:17:58,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4496 >2012-12-17 08:17:58,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4496, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47052de0 >2012-12-17 08:17:58,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling started: taskId = d30aa432-6e3a-4984-b1b2-7ce87cdd91cb >2012-12-17 08:17:59,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling ended: taskId = d30aa432-6e3a-4984-b1b2-7ce87cdd91cb task status = finished >2012-12-17 08:17:59,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] spmStart polling ended, spm status: SPM >2012-12-17 08:17:59,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [60181568] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d30aa432-6e3a-4984-b1b2-7ce87cdd91cb), log id: 494effba >2012-12-17 08:17:59,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [60181568] FINISH, HSMClearTaskVDSCommand, log id: 494effba >2012-12-17 08:17:59,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [60181568] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@617731ef, log id: 47052de0 >2012-12-17 08:17:59,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:17:59,824 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:17:59,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73ef9045 >2012-12-17 08:17:59,834 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [60181568] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:17:59,975 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [36fcbeec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:00,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 597519fd >2012-12-17 08:18:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 772227bb >2012-12-17 08:18:00,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:00,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, SpmStopVDSCommand, log id: 772227bb >2012-12-17 08:18:00,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, ResetIrsVDSCommand, log id: 597519fd >2012-12-17 08:18:00,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b4b4032 >2012-12-17 08:18:00,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:00,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b4b4032 >2012-12-17 08:18:01,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:18:01,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4497 >2012-12-17 08:18:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4497, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 186189d8 >2012-12-17 08:18:01,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling started: taskId = 00044d67-86d7-4c1b-b721-39e201da45ec >2012-12-17 08:18:02,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended: taskId = 00044d67-86d7-4c1b-b721-39e201da45ec task status = finished >2012-12-17 08:18:02,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended, spm status: SPM >2012-12-17 08:18:02,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=00044d67-86d7-4c1b-b721-39e201da45ec), log id: 7016235e >2012-12-17 08:18:02,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMClearTaskVDSCommand, log id: 7016235e >2012-12-17 08:18:02,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d6c29cf, log id: 186189d8 >2012-12-17 08:18:02,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:18:02,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:02,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 326ee144 >2012-12-17 08:18:02,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 326ee144 >2012-12-17 08:18:02,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73ef9045 >2012-12-17 08:18:02,260 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:02,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d87513a >2012-12-17 08:18:02,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: f352c8c >2012-12-17 08:18:02,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: f352c8c >2012-12-17 08:18:02,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d87513a >2012-12-17 08:18:02,281 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [1b451d62] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:06,794 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [6e8742f7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:18:11,102 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [54a63bb2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:18:11,274 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [323f195a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:11,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3930151f >2012-12-17 08:18:11,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ecc7ad7 >2012-12-17 08:18:11,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:11,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, SpmStopVDSCommand, log id: ecc7ad7 >2012-12-17 08:18:12,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, ResetIrsVDSCommand, log id: 3930151f >2012-12-17 08:18:12,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [323f195a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6946e694 >2012-12-17 08:18:12,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:12,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [323f195a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6946e694 >2012-12-17 08:18:17,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [6ad64b5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:18:22,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:18:22,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4498 >2012-12-17 08:18:22,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4498, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5469a2fc >2012-12-17 08:18:22,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling started: taskId = 7b9358e5-3985-47c9-aed2-a8ec71a7592a >2012-12-17 08:18:23,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling ended: taskId = 7b9358e5-3985-47c9-aed2-a8ec71a7592a task status = finished >2012-12-17 08:18:23,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] spmStart polling ended, spm status: SPM >2012-12-17 08:18:23,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3725f22] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7b9358e5-3985-47c9-aed2-a8ec71a7592a), log id: 37d7e074 >2012-12-17 08:18:23,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3725f22] FINISH, HSMClearTaskVDSCommand, log id: 37d7e074 >2012-12-17 08:18:23,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3725f22] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dd0fc50, log id: 5469a2fc >2012-12-17 08:18:23,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:18:23,285 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:18:23,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7eb9922a >2012-12-17 08:18:23,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3725f22] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:18:23,403 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [252efffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:23,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b21a1ec >2012-12-17 08:18:23,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 77a247be >2012-12-17 08:18:23,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:23,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, SpmStopVDSCommand, log id: 77a247be >2012-12-17 08:18:23,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, ResetIrsVDSCommand, log id: 1b21a1ec >2012-12-17 08:18:23,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19bc99cf >2012-12-17 08:18:23,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:23,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19bc99cf >2012-12-17 08:18:23,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:18:23,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4499 >2012-12-17 08:18:23,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4499, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7eb16f28 >2012-12-17 08:18:23,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = dde5d09a-b08a-4a50-bb07-d77c5818bc70 >2012-12-17 08:18:24,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = dde5d09a-b08a-4a50-bb07-d77c5818bc70 task status = finished >2012-12-17 08:18:24,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM >2012-12-17 08:18:24,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dde5d09a-b08a-4a50-bb07-d77c5818bc70), log id: 1da8a934 >2012-12-17 08:18:24,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 1da8a934 >2012-12-17 08:18:24,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bfca422, log id: 7eb16f28 >2012-12-17 08:18:24,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:18:24,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:24,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2874d83b >2012-12-17 08:18:24,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2874d83b >2012-12-17 08:18:24,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7eb9922a >2012-12-17 08:18:24,847 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:24,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a13e8d >2012-12-17 08:18:24,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:24,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19c68305 >2012-12-17 08:18:24,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19c68305 >2012-12-17 08:18:24,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a13e8d >2012-12-17 08:18:24,867 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:27,427 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:18:33,728 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4b0ad6c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:18:33,851 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [48aa9372] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:33,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44bfcc46 >2012-12-17 08:18:33,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: a7cd5df >2012-12-17 08:18:33,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:34,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, SpmStopVDSCommand, log id: a7cd5df >2012-12-17 08:18:34,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, ResetIrsVDSCommand, log id: 44bfcc46 >2012-12-17 08:18:34,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 324d359a >2012-12-17 08:18:34,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:34,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 324d359a >2012-12-17 08:18:37,623 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:18:44,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:18:44,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4500 >2012-12-17 08:18:44,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4500, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b72a5b >2012-12-17 08:18:44,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling started: taskId = 77afb68e-2130-49e9-b91f-92d4dfb55140 >2012-12-17 08:18:45,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling ended: taskId = 77afb68e-2130-49e9-b91f-92d4dfb55140 task status = finished >2012-12-17 08:18:45,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] spmStart polling ended, spm status: SPM >2012-12-17 08:18:45,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [69560fec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77afb68e-2130-49e9-b91f-92d4dfb55140), log id: 6b59fb6f >2012-12-17 08:18:45,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [69560fec] FINISH, HSMClearTaskVDSCommand, log id: 6b59fb6f >2012-12-17 08:18:45,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [69560fec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2db5d1ea, log id: 4b72a5b >2012-12-17 08:18:45,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:18:45,888 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:18:45,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 569151c7 >2012-12-17 08:18:45,905 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [69560fec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:18:45,998 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [bb53630] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:46,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 195e4cb >2012-12-17 08:18:46,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 730a7c60 >2012-12-17 08:18:46,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:46,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, SpmStopVDSCommand, log id: 730a7c60 >2012-12-17 08:18:46,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, ResetIrsVDSCommand, log id: 195e4cb >2012-12-17 08:18:46,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [bb53630] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c64c6fd >2012-12-17 08:18:46,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:46,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [bb53630] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c64c6fd >2012-12-17 08:18:46,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:18:46,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4501 >2012-12-17 08:18:46,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4501, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65d4d291 >2012-12-17 08:18:46,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling started: taskId = 1dd19294-b642-4a31-b11a-77ec6642cb44 >2012-12-17 08:18:47,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended: taskId = 1dd19294-b642-4a31-b11a-77ec6642cb44 task status = finished >2012-12-17 08:18:47,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended, spm status: SPM >2012-12-17 08:18:47,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dd19294-b642-4a31-b11a-77ec6642cb44), log id: 2b9e2bf2 >2012-12-17 08:18:47,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMClearTaskVDSCommand, log id: 2b9e2bf2 >2012-12-17 08:18:47,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d950010, log id: 65d4d291 >2012-12-17 08:18:47,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:18:47,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:47,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48f3506a >2012-12-17 08:18:47,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48f3506a >2012-12-17 08:18:47,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 569151c7 >2012-12-17 08:18:47,504 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:47,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f7b3a39 >2012-12-17 08:18:47,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:47,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b958633 >2012-12-17 08:18:47,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b958633 >2012-12-17 08:18:47,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f7b3a39 >2012-12-17 08:18:47,524 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [8c4c643] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:18:47,798 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:18:56,371 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [57ef526a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:18:56,452 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [376f3052] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:18:56,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: e8ff725 >2012-12-17 08:18:56,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 52fcdc9b >2012-12-17 08:18:56,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:56,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, SpmStopVDSCommand, log id: 52fcdc9b >2012-12-17 08:18:56,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, ResetIrsVDSCommand, log id: e8ff725 >2012-12-17 08:18:56,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a50e390 >2012-12-17 08:18:56,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:18:56,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a50e390 >2012-12-17 08:18:58,072 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:19:06,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:06,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4502 >2012-12-17 08:19:06,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4502, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 372cae3f >2012-12-17 08:19:06,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling started: taskId = 799fc251-db3e-4282-be16-5e2692247abf >2012-12-17 08:19:07,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling ended: taskId = 799fc251-db3e-4282-be16-5e2692247abf task status = finished >2012-12-17 08:19:07,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] spmStart polling ended, spm status: SPM >2012-12-17 08:19:07,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=799fc251-db3e-4282-be16-5e2692247abf), log id: 78ae1939 >2012-12-17 08:19:07,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] FINISH, HSMClearTaskVDSCommand, log id: 78ae1939 >2012-12-17 08:19:07,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [4e011c57] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6dfa2b14, log id: 372cae3f >2012-12-17 08:19:07,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:07,982 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:19:07,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e71d21c >2012-12-17 08:19:07,997 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [4e011c57] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:19:08,123 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:19:08,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ceb3778 >2012-12-17 08:19:08,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54a3aafd >2012-12-17 08:19:08,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:08,299 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:19:08,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStopVDSCommand, log id: 54a3aafd >2012-12-17 08:19:08,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, ResetIrsVDSCommand, log id: ceb3778 >2012-12-17 08:19:08,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e73d544 >2012-12-17 08:19:08,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:08,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e73d544 >2012-12-17 08:19:08,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:08,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4503 >2012-12-17 08:19:08,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4503, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6067309e >2012-12-17 08:19:08,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling started: taskId = 7e795a91-e38f-42ee-9fbb-28c4d9e56bba >2012-12-17 08:19:09,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended: taskId = 7e795a91-e38f-42ee-9fbb-28c4d9e56bba task status = finished >2012-12-17 08:19:09,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended, spm status: SPM >2012-12-17 08:19:09,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e795a91-e38f-42ee-9fbb-28c4d9e56bba), log id: 767854a6 >2012-12-17 08:19:09,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMClearTaskVDSCommand, log id: 767854a6 >2012-12-17 08:19:09,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55fd9b0d, log id: 6067309e >2012-12-17 08:19:09,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:09,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:09,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d11f210 >2012-12-17 08:19:09,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d11f210 >2012-12-17 08:19:09,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e71d21c >2012-12-17 08:19:09,576 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:09,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 585d8b6a >2012-12-17 08:19:09,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:09,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21a260bb >2012-12-17 08:19:09,580 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:19:09,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cfa225a >2012-12-17 08:19:09,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21a260bb >2012-12-17 08:19:09,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 585d8b6a >2012-12-17 08:19:09,613 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5b39276a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:09,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:09,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51aa27d8 >2012-12-17 08:19:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51aa27d8 >2012-12-17 08:19:09,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cfa225a >2012-12-17 08:19:09,636 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:09,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4d2f4495 >2012-12-17 08:19:09,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:09,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b8ef8d >2012-12-17 08:19:09,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b8ef8d >2012-12-17 08:19:09,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4d2f4495 >2012-12-17 08:19:09,656 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:18,499 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [55cb11f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:19:18,582 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [53e8b92c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:19:18,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63b0d77d >2012-12-17 08:19:18,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 486ad17a >2012-12-17 08:19:18,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:18,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, SpmStopVDSCommand, log id: 486ad17a >2012-12-17 08:19:18,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, ResetIrsVDSCommand, log id: 63b0d77d >2012-12-17 08:19:18,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 416bd07e >2012-12-17 08:19:18,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:18,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 416bd07e >2012-12-17 08:19:28,671 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:19:28,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:28,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4504 >2012-12-17 08:19:28,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4504, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5542d527 >2012-12-17 08:19:28,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling started: taskId = c1d4d70f-ad84-4b12-a6fa-a69438848a28 >2012-12-17 08:19:30,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling ended: taskId = c1d4d70f-ad84-4b12-a6fa-a69438848a28 task status = finished >2012-12-17 08:19:30,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] spmStart polling ended, spm status: SPM >2012-12-17 08:19:30,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c1d4d70f-ad84-4b12-a6fa-a69438848a28), log id: 23550266 >2012-12-17 08:19:30,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] FINISH, HSMClearTaskVDSCommand, log id: 23550266 >2012-12-17 08:19:30,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [7d2c68ac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@189b2c00, log id: 5542d527 >2012-12-17 08:19:30,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:31,343 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:19:31,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7467f20f >2012-12-17 08:19:31,344 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [7d2c68ac] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:19:31,893 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [76a14bbb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:19:31,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: fc5e015 >2012-12-17 08:19:31,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 635b13a8 >2012-12-17 08:19:31,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:32,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SpmStopVDSCommand, log id: 635b13a8 >2012-12-17 08:19:32,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, ResetIrsVDSCommand, log id: fc5e015 >2012-12-17 08:19:32,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d6c751b >2012-12-17 08:19:32,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:32,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d6c751b >2012-12-17 08:19:32,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:32,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4505 >2012-12-17 08:19:32,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4505, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e78bcb4 >2012-12-17 08:19:32,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling started: taskId = edaf8c0a-b100-44ad-b86c-216eeb97a6da >2012-12-17 08:19:33,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling ended: taskId = edaf8c0a-b100-44ad-b86c-216eeb97a6da task status = finished >2012-12-17 08:19:33,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] spmStart polling ended, spm status: SPM >2012-12-17 08:19:33,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=edaf8c0a-b100-44ad-b86c-216eeb97a6da), log id: 159d5ba1 >2012-12-17 08:19:33,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMClearTaskVDSCommand, log id: 159d5ba1 >2012-12-17 08:19:33,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@deab21d, log id: 7e78bcb4 >2012-12-17 08:19:33,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:33,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:33,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 528c8c1 >2012-12-17 08:19:33,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 528c8c1 >2012-12-17 08:19:33,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7467f20f >2012-12-17 08:19:33,400 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:33,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50b480bb >2012-12-17 08:19:33,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:33,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cb19ea8 >2012-12-17 08:19:33,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cb19ea8 >2012-12-17 08:19:33,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [55210d2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50b480bb >2012-12-17 08:19:33,419 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [55210d2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:38,861 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:19:42,299 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [780d3d0a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:19:42,391 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [2e21b599] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:19:42,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e279467 >2012-12-17 08:19:42,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d55106a >2012-12-17 08:19:42,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:42,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStopVDSCommand, log id: 1d55106a >2012-12-17 08:19:42,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, ResetIrsVDSCommand, log id: 7e279467 >2012-12-17 08:19:42,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b790c61 >2012-12-17 08:19:42,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:42,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b790c61 >2012-12-17 08:19:49,116 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:19:52,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:52,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4506 >2012-12-17 08:19:52,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4506, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 665d1725 >2012-12-17 08:19:52,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling started: taskId = a265ba60-0a3e-4c66-a952-090480a0a481 >2012-12-17 08:19:53,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling ended: taskId = a265ba60-0a3e-4c66-a952-090480a0a481 task status = finished >2012-12-17 08:19:53,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] spmStart polling ended, spm status: SPM >2012-12-17 08:19:53,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a265ba60-0a3e-4c66-a952-090480a0a481), log id: 198b15a0 >2012-12-17 08:19:53,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] FINISH, HSMClearTaskVDSCommand, log id: 198b15a0 >2012-12-17 08:19:53,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [672e17aa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a38c879, log id: 665d1725 >2012-12-17 08:19:53,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:53,878 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:19:53,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35f11e43 >2012-12-17 08:19:53,898 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [672e17aa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:19:54,001 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:19:54,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2dd81042 >2012-12-17 08:19:54,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 723a9f9a >2012-12-17 08:19:54,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:54,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SpmStopVDSCommand, log id: 723a9f9a >2012-12-17 08:19:54,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, ResetIrsVDSCommand, log id: 2dd81042 >2012-12-17 08:19:54,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42432b10 >2012-12-17 08:19:54,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:54,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42432b10 >2012-12-17 08:19:54,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:19:54,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4507 >2012-12-17 08:19:54,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4507, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 91f4e0f >2012-12-17 08:19:54,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = bb045fbf-b8c8-4489-8548-305a7d43608f >2012-12-17 08:19:55,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = bb045fbf-b8c8-4489-8548-305a7d43608f task status = finished >2012-12-17 08:19:55,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM >2012-12-17 08:19:55,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb045fbf-b8c8-4489-8548-305a7d43608f), log id: 487d1d7c >2012-12-17 08:19:55,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 487d1d7c >2012-12-17 08:19:55,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e1add12, log id: 91f4e0f >2012-12-17 08:19:55,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:19:55,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:55,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25bd1346 >2012-12-17 08:19:55,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25bd1346 >2012-12-17 08:19:55,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35f11e43 >2012-12-17 08:19:55,921 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:55,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 203c6c05 >2012-12-17 08:19:55,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:19:55,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e5d9777 >2012-12-17 08:19:55,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e5d9777 >2012-12-17 08:19:55,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 203c6c05 >2012-12-17 08:19:55,942 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:19:59,292 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Checking autorecoverable hosts >2012-12-17 08:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Autorecovering 0 hosts >2012-12-17 08:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Checking autorecoverable hosts done >2012-12-17 08:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [5a4d3cd3] Autorecovering storage domains is disabled, skipping >2012-12-17 08:20:04,372 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [284eeca1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:20:04,485 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [2afeec77] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:20:04,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5c93c2c5 >2012-12-17 08:20:04,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 582f33ac >2012-12-17 08:20:04,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:04,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, SpmStopVDSCommand, log id: 582f33ac >2012-12-17 08:20:04,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, ResetIrsVDSCommand, log id: 5c93c2c5 >2012-12-17 08:20:04,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49140809 >2012-12-17 08:20:04,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:04,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [2afeec77] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49140809 >2012-12-17 08:20:09,518 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:20:14,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:20:14,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4508 >2012-12-17 08:20:14,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4508, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62b69b4 >2012-12-17 08:20:14,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling started: taskId = 11dfe620-2798-4dd3-81ff-548a07d8465c >2012-12-17 08:20:15,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling ended: taskId = 11dfe620-2798-4dd3-81ff-548a07d8465c task status = finished >2012-12-17 08:20:15,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] spmStart polling ended, spm status: SPM >2012-12-17 08:20:15,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=11dfe620-2798-4dd3-81ff-548a07d8465c), log id: 6caf23f3 >2012-12-17 08:20:15,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] FINISH, HSMClearTaskVDSCommand, log id: 6caf23f3 >2012-12-17 08:20:15,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [6c123ee7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5e615e43, log id: 62b69b4 >2012-12-17 08:20:15,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:20:15,983 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:20:15,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7bc476ac >2012-12-17 08:20:16,006 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6c123ee7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:20:16,100 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [cdacbd1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:20:16,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5c86030c >2012-12-17 08:20:16,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dd4de77 >2012-12-17 08:20:16,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:16,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStopVDSCommand, log id: 5dd4de77 >2012-12-17 08:20:16,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, ResetIrsVDSCommand, log id: 5c86030c >2012-12-17 08:20:16,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b6f69fe >2012-12-17 08:20:16,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:16,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b6f69fe >2012-12-17 08:20:16,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:20:16,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4509 >2012-12-17 08:20:16,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4509, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2583dad2 >2012-12-17 08:20:16,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling started: taskId = 8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27 >2012-12-17 08:20:17,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended: taskId = 8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27 task status = finished >2012-12-17 08:20:17,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended, spm status: SPM >2012-12-17 08:20:17,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8da204d2-ad34-4ee7-8ec6-e6f3ff7feb27), log id: 13f5a565 >2012-12-17 08:20:17,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMClearTaskVDSCommand, log id: 13f5a565 >2012-12-17 08:20:17,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@643d941f, log id: 2583dad2 >2012-12-17 08:20:17,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:20:17,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:17,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 260fb82e >2012-12-17 08:20:17,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 260fb82e >2012-12-17 08:20:17,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bc476ac >2012-12-17 08:20:17,881 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:20:17,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5204e787 >2012-12-17 08:20:17,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:17,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41d4e89 >2012-12-17 08:20:17,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41d4e89 >2012-12-17 08:20:17,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5204e787 >2012-12-17 08:20:17,901 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [25ec3656] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:20:19,767 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:20:26,765 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6123283f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:20:26,993 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [6e9d510a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:20:27,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49805ebe >2012-12-17 08:20:27,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d83d4cf >2012-12-17 08:20:27,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:27,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStopVDSCommand, log id: 4d83d4cf >2012-12-17 08:20:27,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, ResetIrsVDSCommand, log id: 49805ebe >2012-12-17 08:20:27,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 422419f9 >2012-12-17 08:20:27,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:27,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 422419f9 >2012-12-17 08:20:29,938 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [2e10036] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:20:37,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:20:37,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4510 >2012-12-17 08:20:37,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4510, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fe4cff7 >2012-12-17 08:20:37,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling started: taskId = 6f8cd004-c2f8-4a5b-9634-3024632a0b81 >2012-12-17 08:20:39,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling ended: taskId = 6f8cd004-c2f8-4a5b-9634-3024632a0b81 task status = finished >2012-12-17 08:20:39,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] spmStart polling ended, spm status: SPM >2012-12-17 08:20:39,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [508b37e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f8cd004-c2f8-4a5b-9634-3024632a0b81), log id: 633661c7 >2012-12-17 08:20:39,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [508b37e] FINISH, HSMClearTaskVDSCommand, log id: 633661c7 >2012-12-17 08:20:39,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [508b37e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74d82b4, log id: 1fe4cff7 >2012-12-17 08:20:39,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:20:39,561 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:20:39,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6896289e >2012-12-17 08:20:39,573 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [508b37e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:20:39,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [3ae71176] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:20:39,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12ccb585 >2012-12-17 08:20:39,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53bcdb57 >2012-12-17 08:20:39,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:39,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, SpmStopVDSCommand, log id: 53bcdb57 >2012-12-17 08:20:39,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, ResetIrsVDSCommand, log id: 12ccb585 >2012-12-17 08:20:39,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bfaad51 >2012-12-17 08:20:39,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:39,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bfaad51 >2012-12-17 08:20:40,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:20:40,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4511 >2012-12-17 08:20:40,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4511, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 303165ae >2012-12-17 08:20:40,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling started: taskId = c93e1d07-c44f-4931-b41b-11ec50f680d3 >2012-12-17 08:20:41,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended: taskId = c93e1d07-c44f-4931-b41b-11ec50f680d3 task status = finished >2012-12-17 08:20:41,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended, spm status: SPM >2012-12-17 08:20:41,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c93e1d07-c44f-4931-b41b-11ec50f680d3), log id: 6d721fbc >2012-12-17 08:20:41,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMClearTaskVDSCommand, log id: 6d721fbc >2012-12-17 08:20:41,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f81a7c6, log id: 303165ae >2012-12-17 08:20:41,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:20:41,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:41,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: de7e2b8 >2012-12-17 08:20:41,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: de7e2b8 >2012-12-17 08:20:41,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6896289e >2012-12-17 08:20:41,374 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:20:41,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72d5a529 >2012-12-17 08:20:41,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:41,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19a8d474 >2012-12-17 08:20:41,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19a8d474 >2012-12-17 08:20:41,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72d5a529 >2012-12-17 08:20:41,395 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [555e5948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:20:50,089 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [42c50c6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:20:51,477 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [3116bd71] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:20:51,514 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [7582e2ab] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:20:51,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7de310d7 >2012-12-17 08:20:51,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a56a777 >2012-12-17 08:20:51,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:51,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStopVDSCommand, log id: 3a56a777 >2012-12-17 08:20:51,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, ResetIrsVDSCommand, log id: 7de310d7 >2012-12-17 08:20:51,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25ec8691 >2012-12-17 08:20:51,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:20:51,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25ec8691 >2012-12-17 08:21:01,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:02,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4512 >2012-12-17 08:21:02,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4512, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38960148 >2012-12-17 08:21:02,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling started: taskId = f13a806d-10f4-4625-961d-b64eba34017e >2012-12-17 08:21:03,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling ended: taskId = f13a806d-10f4-4625-961d-b64eba34017e task status = finished >2012-12-17 08:21:03,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] spmStart polling ended, spm status: SPM >2012-12-17 08:21:03,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f13a806d-10f4-4625-961d-b64eba34017e), log id: 2c39f0c4 >2012-12-17 08:21:03,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, HSMClearTaskVDSCommand, log id: 2c39f0c4 >2012-12-17 08:21:03,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [16233e3c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@fedea8b, log id: 38960148 >2012-12-17 08:21:03,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:21:03,485 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [16233e3c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:21:03,648 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [42bc2eb6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:21:03,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 60342cfd >2012-12-17 08:21:03,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 66112860 >2012-12-17 08:21:03,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:03,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, SpmStopVDSCommand, log id: 66112860 >2012-12-17 08:21:03,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, ResetIrsVDSCommand, log id: 60342cfd >2012-12-17 08:21:03,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75f8fa48 >2012-12-17 08:21:03,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [42bc2eb6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:03,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [42bc2eb6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75f8fa48 >2012-12-17 08:21:12,570 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:21:13,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:14,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4513 >2012-12-17 08:21:14,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4513, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7968eb0f >2012-12-17 08:21:14,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling started: taskId = 09f13bd1-7812-4e76-874d-5fd46ad2cc87 >2012-12-17 08:21:15,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended: taskId = 09f13bd1-7812-4e76-874d-5fd46ad2cc87 task status = finished >2012-12-17 08:21:15,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] spmStart polling ended, spm status: SPM >2012-12-17 08:21:15,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=09f13bd1-7812-4e76-874d-5fd46ad2cc87), log id: 4d0cc38a >2012-12-17 08:21:15,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, HSMClearTaskVDSCommand, log id: 4d0cc38a >2012-12-17 08:21:15,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1b451d62] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f9c08b, log id: 7968eb0f >2012-12-17 08:21:15,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:21:15,136 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:21:15,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2eb08483 >2012-12-17 08:21:15,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1b451d62] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:21:16,235 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [7bc37769] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:21:16,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2dd6cf7d >2012-12-17 08:21:16,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f79fd76 >2012-12-17 08:21:16,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:16,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, SpmStopVDSCommand, log id: 4f79fd76 >2012-12-17 08:21:16,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, ResetIrsVDSCommand, log id: 2dd6cf7d >2012-12-17 08:21:16,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63f28439 >2012-12-17 08:21:16,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:16,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63f28439 >2012-12-17 08:21:16,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:16,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4514 >2012-12-17 08:21:16,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4514, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e5636b4 >2012-12-17 08:21:16,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling started: taskId = 156fb605-d70d-4649-9872-7123dd9a2c6b >2012-12-17 08:21:17,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended: taskId = 156fb605-d70d-4649-9872-7123dd9a2c6b task status = finished >2012-12-17 08:21:17,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended, spm status: SPM >2012-12-17 08:21:17,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=156fb605-d70d-4649-9872-7123dd9a2c6b), log id: 5cc6900c >2012-12-17 08:21:17,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMClearTaskVDSCommand, log id: 5cc6900c >2012-12-17 08:21:17,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@510dca02, log id: 5e5636b4 >2012-12-17 08:21:17,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:21:17,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:17,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 50d0c5e4 >2012-12-17 08:21:17,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50d0c5e4 >2012-12-17 08:21:17,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2eb08483 >2012-12-17 08:21:17,897 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:21:17,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42cc0ff9 >2012-12-17 08:21:17,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:17,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f7cbbc3 >2012-12-17 08:21:17,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f7cbbc3 >2012-12-17 08:21:17,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42cc0ff9 >2012-12-17 08:21:17,918 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [6e9d510a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:21:22,766 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:21:26,804 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [6ad64b5d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:21:26,964 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [1852110b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:21:27,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7f7dfad1 >2012-12-17 08:21:27,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ad24ff7 >2012-12-17 08:21:27,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:27,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, SpmStopVDSCommand, log id: 3ad24ff7 >2012-12-17 08:21:27,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, ResetIrsVDSCommand, log id: 7f7dfad1 >2012-12-17 08:21:27,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c053df8 >2012-12-17 08:21:27,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:27,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c053df8 >2012-12-17 08:21:32,978 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:21:37,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:37,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4515 >2012-12-17 08:21:37,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4515, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 903f6df >2012-12-17 08:21:37,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling started: taskId = 97bb6416-275f-422f-bf1e-90a9c732f39e >2012-12-17 08:21:38,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended: taskId = 97bb6416-275f-422f-bf1e-90a9c732f39e task status = finished >2012-12-17 08:21:38,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] spmStart polling ended, spm status: SPM >2012-12-17 08:21:38,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=97bb6416-275f-422f-bf1e-90a9c732f39e), log id: 3f6680fd >2012-12-17 08:21:38,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, HSMClearTaskVDSCommand, log id: 3f6680fd >2012-12-17 08:21:38,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@629e828f, log id: 903f6df >2012-12-17 08:21:38,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:21:38,974 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:21:38,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e0d0279 >2012-12-17 08:21:38,994 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a4d3cd3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:21:39,127 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [495d106] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:21:39,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62a575d1 >2012-12-17 08:21:39,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2c177f6b >2012-12-17 08:21:39,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:39,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SpmStopVDSCommand, log id: 2c177f6b >2012-12-17 08:21:39,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, ResetIrsVDSCommand, log id: 62a575d1 >2012-12-17 08:21:39,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a6f3e4c >2012-12-17 08:21:39,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:39,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a6f3e4c >2012-12-17 08:21:39,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:39,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4516 >2012-12-17 08:21:39,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4516, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: aa0426f >2012-12-17 08:21:39,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling started: taskId = 1a3e6f5e-870f-4273-a826-915d53fb4180 >2012-12-17 08:21:40,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling ended: taskId = 1a3e6f5e-870f-4273-a826-915d53fb4180 task status = finished >2012-12-17 08:21:40,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] spmStart polling ended, spm status: SPM >2012-12-17 08:21:40,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1a3e6f5e-870f-4273-a826-915d53fb4180), log id: 7cc18446 >2012-12-17 08:21:40,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMClearTaskVDSCommand, log id: 7cc18446 >2012-12-17 08:21:40,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fe0b098, log id: aa0426f >2012-12-17 08:21:40,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:21:41,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:41,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a400844 >2012-12-17 08:21:41,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a400844 >2012-12-17 08:21:41,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e0d0279 >2012-12-17 08:21:41,390 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:21:41,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31dece71 >2012-12-17 08:21:41,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:41,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f6eeb1f >2012-12-17 08:21:41,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f6eeb1f >2012-12-17 08:21:41,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7de89d45] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31dece71 >2012-12-17 08:21:41,410 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7de89d45] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:21:43,172 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [3e88c176] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:21:49,479 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [219a053f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:21:49,652 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [d425472] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:21:49,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e87f2a7 >2012-12-17 08:21:49,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e764c31 >2012-12-17 08:21:49,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:49,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, SpmStopVDSCommand, log id: 2e764c31 >2012-12-17 08:21:49,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, ResetIrsVDSCommand, log id: 7e87f2a7 >2012-12-17 08:21:49,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b42a617 >2012-12-17 08:21:49,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:21:49,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b42a617 >2012-12-17 08:21:53,357 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:21:59,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:21:59,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4517 >2012-12-17 08:21:59,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4517, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b4a7248 >2012-12-17 08:22:00,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling started: taskId = b98c6a60-cdc1-4ed9-b7fc-10ab769441e3 >2012-12-17 08:22:01,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling ended: taskId = b98c6a60-cdc1-4ed9-b7fc-10ab769441e3 task status = finished >2012-12-17 08:22:01,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] spmStart polling ended, spm status: SPM >2012-12-17 08:22:01,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b98c6a60-cdc1-4ed9-b7fc-10ab769441e3), log id: 29817262 >2012-12-17 08:22:01,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] FINISH, HSMClearTaskVDSCommand, log id: 29817262 >2012-12-17 08:22:01,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7a15027e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@717c9537, log id: 5b4a7248 >2012-12-17 08:22:01,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:01,409 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:22:01,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ecbeb53 >2012-12-17 08:22:01,422 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7a15027e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:01,501 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [5970076c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:01,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65159224 >2012-12-17 08:22:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a46f5e1 >2012-12-17 08:22:01,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:01,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, SpmStopVDSCommand, log id: 4a46f5e1 >2012-12-17 08:22:01,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, ResetIrsVDSCommand, log id: 65159224 >2012-12-17 08:22:01,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [5970076c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c01c559 >2012-12-17 08:22:01,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5970076c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:01,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [5970076c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c01c559 >2012-12-17 08:22:01,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:22:01,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4518 >2012-12-17 08:22:01,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4518, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4523fb89 >2012-12-17 08:22:01,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling started: taskId = d1bc0697-1397-4a73-b9fd-bf57312cffc2 >2012-12-17 08:22:02,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended: taskId = d1bc0697-1397-4a73-b9fd-bf57312cffc2 task status = finished >2012-12-17 08:22:02,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] spmStart polling ended, spm status: SPM >2012-12-17 08:22:02,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d1bc0697-1397-4a73-b9fd-bf57312cffc2), log id: 55847e35 >2012-12-17 08:22:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMClearTaskVDSCommand, log id: 55847e35 >2012-12-17 08:22:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d79f56f, log id: 4523fb89 >2012-12-17 08:22:02,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:02,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40de52ee >2012-12-17 08:22:03,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40de52ee >2012-12-17 08:22:03,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ecbeb53 >2012-12-17 08:22:03,025 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:03,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41b9a612 >2012-12-17 08:22:03,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:03,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c33a992 >2012-12-17 08:22:03,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c33a992 >2012-12-17 08:22:03,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [71ec072c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41b9a612 >2012-12-17 08:22:03,047 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [71ec072c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:03,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:22:11,873 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [f4188d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:12,051 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [778f4477] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:12,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b860d90 >2012-12-17 08:22:12,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3fae04cb >2012-12-17 08:22:12,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:12,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, SpmStopVDSCommand, log id: 3fae04cb >2012-12-17 08:22:12,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, ResetIrsVDSCommand, log id: 1b860d90 >2012-12-17 08:22:12,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [778f4477] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 52f70926 >2012-12-17 08:22:12,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:12,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [778f4477] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 52f70926 >2012-12-17 08:22:13,903 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:22:22,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:22:22,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4519 >2012-12-17 08:22:22,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4519, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5dc7fbc5 >2012-12-17 08:22:22,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling started: taskId = d8e84330-08fe-41e3-bcc4-f271934a58f3 >2012-12-17 08:22:23,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling ended: taskId = d8e84330-08fe-41e3-bcc4-f271934a58f3 task status = finished >2012-12-17 08:22:23,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] spmStart polling ended, spm status: SPM >2012-12-17 08:22:23,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [252efffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8e84330-08fe-41e3-bcc4-f271934a58f3), log id: 3c36b8eb >2012-12-17 08:22:23,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, HSMClearTaskVDSCommand, log id: 3c36b8eb >2012-12-17 08:22:23,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [252efffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ca66b2f, log id: 5dc7fbc5 >2012-12-17 08:22:23,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:24,073 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:22:24,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 61abb047 >2012-12-17 08:22:24,096 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [252efffb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:24,239 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [24da4704] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:24,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a4f8fa4 >2012-12-17 08:22:24,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3719d455 >2012-12-17 08:22:24,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:24,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, SpmStopVDSCommand, log id: 3719d455 >2012-12-17 08:22:24,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, ResetIrsVDSCommand, log id: 7a4f8fa4 >2012-12-17 08:22:24,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [24da4704] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3843543e >2012-12-17 08:22:24,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [24da4704] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:24,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [24da4704] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3843543e >2012-12-17 08:22:24,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:22:24,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4520 >2012-12-17 08:22:24,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4520, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b232817 >2012-12-17 08:22:24,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling started: taskId = adfea206-6d73-4cb9-b86a-e2aaba8016ec >2012-12-17 08:22:25,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended: taskId = adfea206-6d73-4cb9-b86a-e2aaba8016ec task status = finished >2012-12-17 08:22:25,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended, spm status: SPM >2012-12-17 08:22:25,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=adfea206-6d73-4cb9-b86a-e2aaba8016ec), log id: 373b4882 >2012-12-17 08:22:25,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMClearTaskVDSCommand, log id: 373b4882 >2012-12-17 08:22:25,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@15156b22, log id: 5b232817 >2012-12-17 08:22:25,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:25,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:25,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 70926669 >2012-12-17 08:22:25,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 70926669 >2012-12-17 08:22:25,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 61abb047 >2012-12-17 08:22:25,895 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:25,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f9ae83f >2012-12-17 08:22:25,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:25,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 53418b2 >2012-12-17 08:22:25,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 53418b2 >2012-12-17 08:22:25,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f9ae83f >2012-12-17 08:22:25,915 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [43f46040] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:34,306 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:22:34,681 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [667bec9e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:34,765 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [436410e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:34,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67d44a25 >2012-12-17 08:22:34,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44976e8b >2012-12-17 08:22:34,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:35,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SpmStopVDSCommand, log id: 44976e8b >2012-12-17 08:22:35,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, ResetIrsVDSCommand, log id: 67d44a25 >2012-12-17 08:22:35,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e6d1939 >2012-12-17 08:22:35,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:35,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e6d1939 >2012-12-17 08:22:44,532 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:22:45,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:22:45,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4521 >2012-12-17 08:22:45,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4521, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 136c5335 >2012-12-17 08:22:45,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling started: taskId = a7da3510-78bc-44d2-8508-e195d3283129 >2012-12-17 08:22:46,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling ended: taskId = a7da3510-78bc-44d2-8508-e195d3283129 task status = finished >2012-12-17 08:22:46,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] spmStart polling ended, spm status: SPM >2012-12-17 08:22:46,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [1879810a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7da3510-78bc-44d2-8508-e195d3283129), log id: 69273a85 >2012-12-17 08:22:46,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [1879810a] FINISH, HSMClearTaskVDSCommand, log id: 69273a85 >2012-12-17 08:22:46,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [1879810a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e7327b2, log id: 136c5335 >2012-12-17 08:22:46,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:46,494 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:22:46,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3be7b4b8 >2012-12-17 08:22:46,518 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1879810a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:46,651 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [2c13a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:46,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4294af43 >2012-12-17 08:22:46,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f8e1174 >2012-12-17 08:22:46,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:46,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, SpmStopVDSCommand, log id: 1f8e1174 >2012-12-17 08:22:46,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, ResetIrsVDSCommand, log id: 4294af43 >2012-12-17 08:22:46,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 663521de >2012-12-17 08:22:46,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:46,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 663521de >2012-12-17 08:22:46,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:22:46,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4522 >2012-12-17 08:22:46,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4522, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38b6b9b1 >2012-12-17 08:22:46,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling started: taskId = 38cbb08f-4377-489f-bff6-742897a2a3a4 >2012-12-17 08:22:48,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended: taskId = 38cbb08f-4377-489f-bff6-742897a2a3a4 task status = finished >2012-12-17 08:22:48,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended, spm status: SPM >2012-12-17 08:22:48,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38cbb08f-4377-489f-bff6-742897a2a3a4), log id: 8ab4d1 >2012-12-17 08:22:48,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMClearTaskVDSCommand, log id: 8ab4d1 >2012-12-17 08:22:48,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1265059e, log id: 38b6b9b1 >2012-12-17 08:22:48,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:22:48,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:48,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 269af953 >2012-12-17 08:22:48,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 269af953 >2012-12-17 08:22:48,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3be7b4b8 >2012-12-17 08:22:48,109 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:48,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 866cbf2 >2012-12-17 08:22:48,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:48,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b7e58da >2012-12-17 08:22:48,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b7e58da >2012-12-17 08:22:48,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 866cbf2 >2012-12-17 08:22:48,129 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [7543fd1a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:22:54,740 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:22:56,994 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [44853924] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:22:57,264 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:22:57,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1effcdb1 >2012-12-17 08:22:57,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41f10a40 >2012-12-17 08:22:57,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:57,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStopVDSCommand, log id: 41f10a40 >2012-12-17 08:22:57,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, ResetIrsVDSCommand, log id: 1effcdb1 >2012-12-17 08:22:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8a6a68a >2012-12-17 08:22:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:22:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8a6a68a >2012-12-17 08:23:04,958 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:23:07,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:07,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4523 >2012-12-17 08:23:07,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4523, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cf31323 >2012-12-17 08:23:07,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling started: taskId = a0ef1178-c3fc-493d-afa6-e7fdfe352440 >2012-12-17 08:23:08,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended: taskId = a0ef1178-c3fc-493d-afa6-e7fdfe352440 task status = finished >2012-12-17 08:23:08,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] spmStart polling ended, spm status: SPM >2012-12-17 08:23:08,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a0ef1178-c3fc-493d-afa6-e7fdfe352440), log id: 603fb910 >2012-12-17 08:23:08,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, HSMClearTaskVDSCommand, log id: 603fb910 >2012-12-17 08:23:08,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5b39276a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@778efacd, log id: 6cf31323 >2012-12-17 08:23:08,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:08,831 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:23:08,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77017160 >2012-12-17 08:23:08,853 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5b39276a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:23:09,036 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [723e6c9b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:23:09,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5332a348 >2012-12-17 08:23:09,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6482688b >2012-12-17 08:23:09,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:09,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, SpmStopVDSCommand, log id: 6482688b >2012-12-17 08:23:09,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, ResetIrsVDSCommand, log id: 5332a348 >2012-12-17 08:23:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 453e65d5 >2012-12-17 08:23:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [723e6c9b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [723e6c9b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 453e65d5 >2012-12-17 08:23:09,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:09,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4524 >2012-12-17 08:23:09,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4524, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 303db02a >2012-12-17 08:23:09,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling started: taskId = fe5a8e84-c373-42f3-aa6a-170cbc3bce31 >2012-12-17 08:23:10,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended: taskId = fe5a8e84-c373-42f3-aa6a-170cbc3bce31 task status = finished >2012-12-17 08:23:10,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended, spm status: SPM >2012-12-17 08:23:10,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fe5a8e84-c373-42f3-aa6a-170cbc3bce31), log id: 3f0f366b >2012-12-17 08:23:10,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMClearTaskVDSCommand, log id: 3f0f366b >2012-12-17 08:23:10,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fbdcb24, log id: 303db02a >2012-12-17 08:23:11,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:11,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:11,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ab85557 >2012-12-17 08:23:11,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ab85557 >2012-12-17 08:23:11,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77017160 >2012-12-17 08:23:11,139 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:23:11,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6beaf97d >2012-12-17 08:23:11,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:11,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49689b4 >2012-12-17 08:23:11,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49689b4 >2012-12-17 08:23:11,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6beaf97d >2012-12-17 08:23:11,158 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [1143d064] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:23:15,896 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:23:19,917 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [9dcd5cb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:23:20,019 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [43662b97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:23:20,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 538b4e92 >2012-12-17 08:23:20,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4774a893 >2012-12-17 08:23:20,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:20,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, SpmStopVDSCommand, log id: 4774a893 >2012-12-17 08:23:20,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, ResetIrsVDSCommand, log id: 538b4e92 >2012-12-17 08:23:20,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63e0791e >2012-12-17 08:23:20,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:20,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63e0791e >2012-12-17 08:23:26,501 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:23:31,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:31,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4525 >2012-12-17 08:23:31,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4525, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11908b9 >2012-12-17 08:23:31,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling started: taskId = 7e4f0d30-d60b-4057-9e50-2c6109092f41 >2012-12-17 08:23:32,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling ended: taskId = 7e4f0d30-d60b-4057-9e50-2c6109092f41 task status = finished >2012-12-17 08:23:32,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] spmStart polling ended, spm status: SPM >2012-12-17 08:23:32,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e4f0d30-d60b-4057-9e50-2c6109092f41), log id: 5d813997 >2012-12-17 08:23:32,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] FINISH, HSMClearTaskVDSCommand, log id: 5d813997 >2012-12-17 08:23:32,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4eec1876, log id: 11908b9 >2012-12-17 08:23:32,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:32,448 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:23:32,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 589c6b47 >2012-12-17 08:23:32,468 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [7c9ffe1c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:23:32,608 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [50c114a6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:23:32,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a111bde >2012-12-17 08:23:32,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6401f4e3 >2012-12-17 08:23:32,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:33,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, SpmStopVDSCommand, log id: 6401f4e3 >2012-12-17 08:23:33,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, ResetIrsVDSCommand, log id: 1a111bde >2012-12-17 08:23:33,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61f0879b >2012-12-17 08:23:33,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:33,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [50c114a6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61f0879b >2012-12-17 08:23:33,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:33,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4526 >2012-12-17 08:23:33,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4526, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39f86d1d >2012-12-17 08:23:33,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling started: taskId = c6631b08-7a9f-4bbd-8753-8c670417cece >2012-12-17 08:23:34,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended: taskId = c6631b08-7a9f-4bbd-8753-8c670417cece task status = finished >2012-12-17 08:23:34,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended, spm status: SPM >2012-12-17 08:23:34,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6631b08-7a9f-4bbd-8753-8c670417cece), log id: 2814fab6 >2012-12-17 08:23:34,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMClearTaskVDSCommand, log id: 2814fab6 >2012-12-17 08:23:34,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7957d748, log id: 39f86d1d >2012-12-17 08:23:34,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:34,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:34,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 383b748 >2012-12-17 08:23:34,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 383b748 >2012-12-17 08:23:34,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 589c6b47 >2012-12-17 08:23:34,720 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:23:34,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c75d9c5 >2012-12-17 08:23:34,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:34,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b5ee868 >2012-12-17 08:23:34,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b5ee868 >2012-12-17 08:23:34,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c75d9c5 >2012-12-17 08:23:34,740 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:23:36,820 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:23:43,545 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [51190a74] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:23:43,638 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [6462735f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:23:43,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 699f1afc >2012-12-17 08:23:43,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a5022c4 >2012-12-17 08:23:43,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:43,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, SpmStopVDSCommand, log id: 4a5022c4 >2012-12-17 08:23:43,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, ResetIrsVDSCommand, log id: 699f1afc >2012-12-17 08:23:43,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 157372b7 >2012-12-17 08:23:43,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:43,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 157372b7 >2012-12-17 08:23:47,005 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:23:53,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:53,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4527 >2012-12-17 08:23:53,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4527, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67a704ca >2012-12-17 08:23:53,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling started: taskId = 20bb434f-039e-486a-b697-abd2cf23a7ea >2012-12-17 08:23:55,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended: taskId = 20bb434f-039e-486a-b697-abd2cf23a7ea task status = finished >2012-12-17 08:23:55,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] spmStart polling ended, spm status: SPM >2012-12-17 08:23:55,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20bb434f-039e-486a-b697-abd2cf23a7ea), log id: 16b8e7cf >2012-12-17 08:23:55,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, HSMClearTaskVDSCommand, log id: 16b8e7cf >2012-12-17 08:23:55,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [555e5948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@457ad2b0, log id: 67a704ca >2012-12-17 08:23:55,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:55,148 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:23:55,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d964b71 >2012-12-17 08:23:55,629 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [555e5948] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:23:56,342 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [3bf47b4b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:23:56,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62d5b49b >2012-12-17 08:23:56,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7807218c >2012-12-17 08:23:56,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:56,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, SpmStopVDSCommand, log id: 7807218c >2012-12-17 08:23:56,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, ResetIrsVDSCommand, log id: 62d5b49b >2012-12-17 08:23:56,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ce9f4c3 >2012-12-17 08:23:56,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3bf47b4b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:56,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3bf47b4b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ce9f4c3 >2012-12-17 08:23:56,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:23:56,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4528 >2012-12-17 08:23:56,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4528, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: eda47cb >2012-12-17 08:23:56,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling started: taskId = fd130412-3c78-45f6-a333-8f6c3d90227f >2012-12-17 08:23:57,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended: taskId = fd130412-3c78-45f6-a333-8f6c3d90227f task status = finished >2012-12-17 08:23:57,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended, spm status: SPM >2012-12-17 08:23:57,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fd130412-3c78-45f6-a333-8f6c3d90227f), log id: 28d9ac6a >2012-12-17 08:23:57,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMClearTaskVDSCommand, log id: 28d9ac6a >2012-12-17 08:23:57,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ac67819, log id: eda47cb >2012-12-17 08:23:57,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:23:57,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:57,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71b7adeb >2012-12-17 08:23:57,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71b7adeb >2012-12-17 08:23:57,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d964b71 >2012-12-17 08:23:57,978 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:23:57,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e901274 >2012-12-17 08:23:57,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:23:57,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d4377c0 >2012-12-17 08:23:58,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d4377c0 >2012-12-17 08:23:58,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e901274 >2012-12-17 08:23:58,021 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:24:06,761 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [3e88c176] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:24:06,889 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:24:06,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: f2c7639 >2012-12-17 08:24:06,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c117069 >2012-12-17 08:24:06,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:07,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SpmStopVDSCommand, log id: 3c117069 >2012-12-17 08:24:07,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, ResetIrsVDSCommand, log id: f2c7639 >2012-12-17 08:24:07,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3436f61f >2012-12-17 08:24:07,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:07,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3436f61f >2012-12-17 08:24:07,472 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:24:17,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:24:17,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4529 >2012-12-17 08:24:17,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4529, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 583dcef7 >2012-12-17 08:24:17,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling started: taskId = 8670f109-0a2a-43db-9616-56ce935f424d >2012-12-17 08:24:18,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling ended: taskId = 8670f109-0a2a-43db-9616-56ce935f424d task status = finished >2012-12-17 08:24:18,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] spmStart polling ended, spm status: SPM >2012-12-17 08:24:18,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8670f109-0a2a-43db-9616-56ce935f424d), log id: 1e615734 >2012-12-17 08:24:18,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] FINISH, HSMClearTaskVDSCommand, log id: 1e615734 >2012-12-17 08:24:18,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [33a3b92a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8a42681, log id: 583dcef7 >2012-12-17 08:24:18,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:24:18,345 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:24:18,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29aede03 >2012-12-17 08:24:18,390 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [33a3b92a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:24:18,527 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [783fb511] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:24:18,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5fc69a3c >2012-12-17 08:24:18,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34e029a0 >2012-12-17 08:24:18,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:18,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SpmStopVDSCommand, log id: 34e029a0 >2012-12-17 08:24:18,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, ResetIrsVDSCommand, log id: 5fc69a3c >2012-12-17 08:24:18,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70d3e5d3 >2012-12-17 08:24:18,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:18,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70d3e5d3 >2012-12-17 08:24:18,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:24:18,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4530 >2012-12-17 08:24:18,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4530, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 571879c4 >2012-12-17 08:24:18,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling started: taskId = 9b34cb2b-e500-46d0-b812-724460d47989 >2012-12-17 08:24:19,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling ended: taskId = 9b34cb2b-e500-46d0-b812-724460d47989 task status = finished >2012-12-17 08:24:19,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] spmStart polling ended, spm status: SPM >2012-12-17 08:24:19,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b34cb2b-e500-46d0-b812-724460d47989), log id: 55121dec >2012-12-17 08:24:19,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMClearTaskVDSCommand, log id: 55121dec >2012-12-17 08:24:19,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@186a9c96, log id: 571879c4 >2012-12-17 08:24:19,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:24:19,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:19,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 349124d9 >2012-12-17 08:24:19,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 349124d9 >2012-12-17 08:24:19,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29aede03 >2012-12-17 08:24:19,955 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:24:19,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 250a3d85 >2012-12-17 08:24:19,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:19,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d9d8b27 >2012-12-17 08:24:19,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d9d8b27 >2012-12-17 08:24:19,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [381e4566] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 250a3d85 >2012-12-17 08:24:19,974 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [381e4566] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:24:27,865 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:24:28,877 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [2e10036] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:24:29,065 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4579c6a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:24:29,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9fa5082 >2012-12-17 08:24:29,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1158e3f6 >2012-12-17 08:24:29,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:29,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, SpmStopVDSCommand, log id: 1158e3f6 >2012-12-17 08:24:29,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, ResetIrsVDSCommand, log id: 9fa5082 >2012-12-17 08:24:29,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36766928 >2012-12-17 08:24:29,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:29,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4579c6a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36766928 >2012-12-17 08:24:38,072 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:24:39,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:24:39,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4531 >2012-12-17 08:24:39,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4531, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799c8a86 >2012-12-17 08:24:39,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling started: taskId = 8ca44242-b464-4775-bd44-a21c1db24ac2 >2012-12-17 08:24:40,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling ended: taskId = 8ca44242-b464-4775-bd44-a21c1db24ac2 task status = finished >2012-12-17 08:24:40,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] spmStart polling ended, spm status: SPM >2012-12-17 08:24:40,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ca44242-b464-4775-bd44-a21c1db24ac2), log id: 671eff28 >2012-12-17 08:24:40,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] FINISH, HSMClearTaskVDSCommand, log id: 671eff28 >2012-12-17 08:24:40,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [3940cca7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2401cffc, log id: 799c8a86 >2012-12-17 08:24:40,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:24:41,393 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:24:41,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cb0a668 >2012-12-17 08:24:41,718 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3940cca7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:24:42,014 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [57b06faf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:24:42,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a8177d2 >2012-12-17 08:24:42,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e67524c >2012-12-17 08:24:42,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:42,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, SpmStopVDSCommand, log id: 3e67524c >2012-12-17 08:24:42,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, ResetIrsVDSCommand, log id: a8177d2 >2012-12-17 08:24:42,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 432c6d65 >2012-12-17 08:24:42,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [57b06faf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:42,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [57b06faf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 432c6d65 >2012-12-17 08:24:42,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:24:42,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4532 >2012-12-17 08:24:42,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4532, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 492a24ea >2012-12-17 08:24:42,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling started: taskId = ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad >2012-12-17 08:24:43,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended: taskId = ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad task status = finished >2012-12-17 08:24:43,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended, spm status: SPM >2012-12-17 08:24:43,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ff5e9ead-4bc7-4a9a-b2ea-dfddfa6240ad), log id: 1b921caa >2012-12-17 08:24:43,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMClearTaskVDSCommand, log id: 1b921caa >2012-12-17 08:24:43,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1597eb0a, log id: 492a24ea >2012-12-17 08:24:43,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:24:43,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:43,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12f0af11 >2012-12-17 08:24:44,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12f0af11 >2012-12-17 08:24:44,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cb0a668 >2012-12-17 08:24:44,022 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:24:44,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77758ecd >2012-12-17 08:24:44,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:44,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e683f16 >2012-12-17 08:24:44,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e683f16 >2012-12-17 08:24:44,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77758ecd >2012-12-17 08:24:44,043 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [5a89a26c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:24:48,274 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:24:52,839 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7de89d45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:24:53,023 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [65976fd0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:24:53,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 525215ea >2012-12-17 08:24:53,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 644cc028 >2012-12-17 08:24:53,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:53,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SpmStopVDSCommand, log id: 644cc028 >2012-12-17 08:24:53,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, ResetIrsVDSCommand, log id: 525215ea >2012-12-17 08:24:53,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f8fad0c >2012-12-17 08:24:53,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:24:53,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f8fad0c >2012-12-17 08:24:58,523 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Checking autorecoverable hosts >2012-12-17 08:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Autorecovering 0 hosts >2012-12-17 08:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Checking autorecoverable hosts done >2012-12-17 08:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [2321a59a] Autorecovering storage domains is disabled, skipping >2012-12-17 08:25:03,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:03,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4533 >2012-12-17 08:25:03,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4533, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 629970ab >2012-12-17 08:25:03,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling started: taskId = d2ca93c5-9cfa-43ff-8e77-48dff8d88a67 >2012-12-17 08:25:04,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling ended: taskId = d2ca93c5-9cfa-43ff-8e77-48dff8d88a67 task status = finished >2012-12-17 08:25:04,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] spmStart polling ended, spm status: SPM >2012-12-17 08:25:04,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2ca93c5-9cfa-43ff-8e77-48dff8d88a67), log id: 7efcc417 >2012-12-17 08:25:04,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] FINISH, HSMClearTaskVDSCommand, log id: 7efcc417 >2012-12-17 08:25:04,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [40098d4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ae0323, log id: 629970ab >2012-12-17 08:25:04,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:04,604 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:25:04,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58639ce6 >2012-12-17 08:25:04,637 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [40098d4e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:25:04,761 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [311c1434] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:25:04,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 73e1730 >2012-12-17 08:25:04,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54f6f5dc >2012-12-17 08:25:04,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:05,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, SpmStopVDSCommand, log id: 54f6f5dc >2012-12-17 08:25:05,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, ResetIrsVDSCommand, log id: 73e1730 >2012-12-17 08:25:05,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [311c1434] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62b258ec >2012-12-17 08:25:05,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:05,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [311c1434] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62b258ec >2012-12-17 08:25:05,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:05,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4534 >2012-12-17 08:25:05,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4534, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54f1b836 >2012-12-17 08:25:05,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 769ad798-7be4-4497-86a3-8594373d3c96 >2012-12-17 08:25:06,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 769ad798-7be4-4497-86a3-8594373d3c96 task status = finished >2012-12-17 08:25:06,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM >2012-12-17 08:25:06,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=769ad798-7be4-4497-86a3-8594373d3c96), log id: 1396e305 >2012-12-17 08:25:06,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 1396e305 >2012-12-17 08:25:06,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1712674f, log id: 54f1b836 >2012-12-17 08:25:06,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:06,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:06,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d49c9e6 >2012-12-17 08:25:06,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d49c9e6 >2012-12-17 08:25:06,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58639ce6 >2012-12-17 08:25:06,247 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:06,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7665318 >2012-12-17 08:25:06,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:06,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ee2225e >2012-12-17 08:25:06,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ee2225e >2012-12-17 08:25:06,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7665318 >2012-12-17 08:25:06,267 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:08,769 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [64fd8799] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:25:15,140 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [6e8742f7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:25:15,827 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [5c1f293f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:25:15,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4d80c5b9 >2012-12-17 08:25:15,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2448544a >2012-12-17 08:25:15,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:16,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, SpmStopVDSCommand, log id: 2448544a >2012-12-17 08:25:16,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, ResetIrsVDSCommand, log id: 4d80c5b9 >2012-12-17 08:25:16,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77f6e099 >2012-12-17 08:25:16,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5c1f293f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:16,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5c1f293f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77f6e099 >2012-12-17 08:25:19,000 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:25:26,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:26,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4535 >2012-12-17 08:25:26,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4535, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70d55415 >2012-12-17 08:25:26,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling started: taskId = eb687825-b1d2-4833-be3f-d99c342899f3 >2012-12-17 08:25:27,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling ended: taskId = eb687825-b1d2-4833-be3f-d99c342899f3 task status = finished >2012-12-17 08:25:27,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] spmStart polling ended, spm status: SPM >2012-12-17 08:25:27,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb687825-b1d2-4833-be3f-d99c342899f3), log id: 793709c5 >2012-12-17 08:25:27,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] FINISH, HSMClearTaskVDSCommand, log id: 793709c5 >2012-12-17 08:25:27,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [6fac8ea3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c62a2af, log id: 70d55415 >2012-12-17 08:25:27,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:27,410 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:25:27,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5dafa7fc >2012-12-17 08:25:27,434 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6fac8ea3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:25:27,554 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:25:27,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ead29d9 >2012-12-17 08:25:27,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 273825e3 >2012-12-17 08:25:27,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:27,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SpmStopVDSCommand, log id: 273825e3 >2012-12-17 08:25:27,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, ResetIrsVDSCommand, log id: ead29d9 >2012-12-17 08:25:27,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 688a5434 >2012-12-17 08:25:27,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:27,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 688a5434 >2012-12-17 08:25:27,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:27,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4536 >2012-12-17 08:25:27,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4536, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6fc1de2d >2012-12-17 08:25:27,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling started: taskId = 69f0ce4d-a1f6-4abb-8516-5fb35fbebc10 >2012-12-17 08:25:28,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling ended: taskId = 69f0ce4d-a1f6-4abb-8516-5fb35fbebc10 task status = finished >2012-12-17 08:25:28,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] spmStart polling ended, spm status: SPM >2012-12-17 08:25:28,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=69f0ce4d-a1f6-4abb-8516-5fb35fbebc10), log id: 5a716c79 >2012-12-17 08:25:29,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMClearTaskVDSCommand, log id: 5a716c79 >2012-12-17 08:25:29,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c88aa52, log id: 6fc1de2d >2012-12-17 08:25:29,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:29,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:29,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d5b238e >2012-12-17 08:25:29,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d5b238e >2012-12-17 08:25:29,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5dafa7fc >2012-12-17 08:25:29,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:29,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2dd6487c >2012-12-17 08:25:29,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:29,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16138d1f >2012-12-17 08:25:29,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16138d1f >2012-12-17 08:25:29,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [6bdbfd31] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2dd6487c >2012-12-17 08:25:29,093 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [6bdbfd31] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:29,243 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [48fde2a8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:25:37,987 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [1a57398] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:25:38,192 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [65c3c04c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:25:38,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a7a58f6 >2012-12-17 08:25:38,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69e8e155 >2012-12-17 08:25:38,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:39,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, SpmStopVDSCommand, log id: 69e8e155 >2012-12-17 08:25:39,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, ResetIrsVDSCommand, log id: 3a7a58f6 >2012-12-17 08:25:39,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49f73630 >2012-12-17 08:25:39,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [65c3c04c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:39,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [65c3c04c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49f73630 >2012-12-17 08:25:39,455 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:25:49,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:49,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4537 >2012-12-17 08:25:49,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4537, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5157b0 >2012-12-17 08:25:49,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling started: taskId = 19e2515e-01f6-41fb-ad2f-80e2266c827b >2012-12-17 08:25:50,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling ended: taskId = 19e2515e-01f6-41fb-ad2f-80e2266c827b task status = finished >2012-12-17 08:25:50,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] spmStart polling ended, spm status: SPM >2012-12-17 08:25:50,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [d425472] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19e2515e-01f6-41fb-ad2f-80e2266c827b), log id: 60e593e7 >2012-12-17 08:25:50,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, HSMClearTaskVDSCommand, log id: 60e593e7 >2012-12-17 08:25:50,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [d425472] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@515464, log id: 5157b0 >2012-12-17 08:25:50,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:51,438 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:25:51,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1bedef9e >2012-12-17 08:25:51,460 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [d425472] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:25:51,762 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [3e6ad503] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:25:51,774 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-57) [55210d2] DbUserCacheManager::refreshAllUserData() - entered >2012-12-17 08:25:51,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d232026 >2012-12-17 08:25:51,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23186edd >2012-12-17 08:25:51,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:52,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, SpmStopVDSCommand, log id: 23186edd >2012-12-17 08:25:52,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, ResetIrsVDSCommand, log id: 3d232026 >2012-12-17 08:25:52,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14287446 >2012-12-17 08:25:52,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:52,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e6ad503] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14287446 >2012-12-17 08:25:52,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:25:52,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4538 >2012-12-17 08:25:52,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4538, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20c3733a >2012-12-17 08:25:52,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling started: taskId = 5859914b-f5c7-4308-9ace-246152dc68a9 >2012-12-17 08:25:53,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling ended: taskId = 5859914b-f5c7-4308-9ace-246152dc68a9 task status = finished >2012-12-17 08:25:53,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] spmStart polling ended, spm status: SPM >2012-12-17 08:25:53,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5859914b-f5c7-4308-9ace-246152dc68a9), log id: 4838c30e >2012-12-17 08:25:53,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMClearTaskVDSCommand, log id: 4838c30e >2012-12-17 08:25:53,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68df12d, log id: 20c3733a >2012-12-17 08:25:53,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:25:53,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:53,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4622bfe0 >2012-12-17 08:25:53,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4622bfe0 >2012-12-17 08:25:53,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1bedef9e >2012-12-17 08:25:53,928 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:53,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54a91c40 >2012-12-17 08:25:53,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:25:53,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 623618cf >2012-12-17 08:25:53,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 623618cf >2012-12-17 08:25:53,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [76a14bbb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54a91c40 >2012-12-17 08:25:53,949 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [76a14bbb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:25:59,989 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:02,790 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5c1f293f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:26:02,940 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [67020b95] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:26:03,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bb0db57 >2012-12-17 08:26:03,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e8e78ea >2012-12-17 08:26:03,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:03,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStopVDSCommand, log id: 6e8e78ea >2012-12-17 08:26:03,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, ResetIrsVDSCommand, log id: 2bb0db57 >2012-12-17 08:26:03,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d2a1b94 >2012-12-17 08:26:03,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:03,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d2a1b94 >2012-12-17 08:26:10,189 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:13,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:26:13,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4539 >2012-12-17 08:26:13,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4539, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ae92d37 >2012-12-17 08:26:13,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling started: taskId = b926e5a5-cc04-4425-82e8-1b783d9deef3 >2012-12-17 08:26:14,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling ended: taskId = b926e5a5-cc04-4425-82e8-1b783d9deef3 task status = finished >2012-12-17 08:26:14,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] spmStart polling ended, spm status: SPM >2012-12-17 08:26:14,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [1852110b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b926e5a5-cc04-4425-82e8-1b783d9deef3), log id: bdb6713 >2012-12-17 08:26:14,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, HSMClearTaskVDSCommand, log id: bdb6713 >2012-12-17 08:26:14,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [1852110b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@714c1679, log id: 5ae92d37 >2012-12-17 08:26:14,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:26:14,603 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:26:14,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b233a27 >2012-12-17 08:26:14,669 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1852110b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:26:14,849 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [5863fdac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:26:14,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32e6638a >2012-12-17 08:26:14,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d226762 >2012-12-17 08:26:14,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:15,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, SpmStopVDSCommand, log id: 6d226762 >2012-12-17 08:26:15,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, ResetIrsVDSCommand, log id: 32e6638a >2012-12-17 08:26:15,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b0cf9a5 >2012-12-17 08:26:15,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:15,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b0cf9a5 >2012-12-17 08:26:15,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:26:16,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4540 >2012-12-17 08:26:16,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4540, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76e139c1 >2012-12-17 08:26:16,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling started: taskId = 57201b7e-e6d7-4554-ba80-8c6cf330f9dd >2012-12-17 08:26:17,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling ended: taskId = 57201b7e-e6d7-4554-ba80-8c6cf330f9dd task status = finished >2012-12-17 08:26:17,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] spmStart polling ended, spm status: SPM >2012-12-17 08:26:17,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57201b7e-e6d7-4554-ba80-8c6cf330f9dd), log id: a105d28 >2012-12-17 08:26:17,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMClearTaskVDSCommand, log id: a105d28 >2012-12-17 08:26:17,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3853f241, log id: 76e139c1 >2012-12-17 08:26:17,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:26:17,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:17,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 771c9e3d >2012-12-17 08:26:17,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 771c9e3d >2012-12-17 08:26:17,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b233a27 >2012-12-17 08:26:17,381 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:26:17,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4473b7e6 >2012-12-17 08:26:17,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:17,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b628257 >2012-12-17 08:26:17,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b628257 >2012-12-17 08:26:17,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [5a98cdcf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4473b7e6 >2012-12-17 08:26:17,402 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [5a98cdcf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:26:20,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:25,379 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [65c3c04c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:26:26,216 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [152503f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:26:26,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 548bed55 >2012-12-17 08:26:26,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23ecef64 >2012-12-17 08:26:26,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:26,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, SpmStopVDSCommand, log id: 23ecef64 >2012-12-17 08:26:26,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, ResetIrsVDSCommand, log id: 548bed55 >2012-12-17 08:26:26,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47c9aeb2 >2012-12-17 08:26:26,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:26,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47c9aeb2 >2012-12-17 08:26:31,620 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:36,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:26:36,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4541 >2012-12-17 08:26:36,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4541, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 514908fd >2012-12-17 08:26:36,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling started: taskId = dbecc6ac-9776-4181-83ce-7673e9e7116e >2012-12-17 08:26:37,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling ended: taskId = dbecc6ac-9776-4181-83ce-7673e9e7116e task status = finished >2012-12-17 08:26:37,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] spmStart polling ended, spm status: SPM >2012-12-17 08:26:37,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dbecc6ac-9776-4181-83ce-7673e9e7116e), log id: 73b02d14 >2012-12-17 08:26:37,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] FINISH, HSMClearTaskVDSCommand, log id: 73b02d14 >2012-12-17 08:26:37,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6995ddd2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6aa261a6, log id: 514908fd >2012-12-17 08:26:37,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:26:37,764 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:26:37,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7df0bb6b >2012-12-17 08:26:37,784 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6995ddd2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:26:37,998 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [1eda0292] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:26:38,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a3a2fc6 >2012-12-17 08:26:38,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 680a596e >2012-12-17 08:26:38,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:38,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, SpmStopVDSCommand, log id: 680a596e >2012-12-17 08:26:38,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, ResetIrsVDSCommand, log id: 6a3a2fc6 >2012-12-17 08:26:38,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4771ecb5 >2012-12-17 08:26:38,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1eda0292] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:38,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1eda0292] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4771ecb5 >2012-12-17 08:26:38,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:26:38,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4542 >2012-12-17 08:26:38,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4542, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40a93a5d >2012-12-17 08:26:38,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = a092db07-2a7c-47ae-a134-137a45cd4354 >2012-12-17 08:26:39,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = a092db07-2a7c-47ae-a134-137a45cd4354 task status = finished >2012-12-17 08:26:39,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM >2012-12-17 08:26:39,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a092db07-2a7c-47ae-a134-137a45cd4354), log id: 6efac562 >2012-12-17 08:26:39,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 6efac562 >2012-12-17 08:26:39,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11b4ddda, log id: 40a93a5d >2012-12-17 08:26:39,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:26:39,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:39,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21eb2a3d >2012-12-17 08:26:39,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21eb2a3d >2012-12-17 08:26:39,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7df0bb6b >2012-12-17 08:26:39,733 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:26:39,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: aa62346 >2012-12-17 08:26:39,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:39,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ce1464 >2012-12-17 08:26:39,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ce1464 >2012-12-17 08:26:39,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: aa62346 >2012-12-17 08:26:39,754 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:26:41,814 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:48,597 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [71ec072c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:26:48,727 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [664081c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:26:48,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71c8c33c >2012-12-17 08:26:48,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 176ba17a >2012-12-17 08:26:48,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:49,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, SpmStopVDSCommand, log id: 176ba17a >2012-12-17 08:26:49,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, ResetIrsVDSCommand, log id: 71c8c33c >2012-12-17 08:26:49,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4150ab26 >2012-12-17 08:26:49,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:26:49,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4150ab26 >2012-12-17 08:26:52,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:26:59,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:26:59,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4543 >2012-12-17 08:26:59,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4543, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7219d0c2 >2012-12-17 08:26:59,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling started: taskId = 5b6b6902-4e29-4047-b177-c66353d712d3 >2012-12-17 08:27:00,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling ended: taskId = 5b6b6902-4e29-4047-b177-c66353d712d3 task status = finished >2012-12-17 08:27:00,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] spmStart polling ended, spm status: SPM >2012-12-17 08:27:00,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5b6b6902-4e29-4047-b177-c66353d712d3), log id: 56f658c2 >2012-12-17 08:27:00,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, HSMClearTaskVDSCommand, log id: 56f658c2 >2012-12-17 08:27:00,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3ae71176] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@473d5528, log id: 7219d0c2 >2012-12-17 08:27:00,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:01,088 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:27:01,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 38729bd7 >2012-12-17 08:27:01,161 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3ae71176] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:01,391 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [50f32e72] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:01,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 534fec55 >2012-12-17 08:27:01,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: decf640 >2012-12-17 08:27:01,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:01,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStopVDSCommand, log id: decf640 >2012-12-17 08:27:01,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, ResetIrsVDSCommand, log id: 534fec55 >2012-12-17 08:27:01,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ef64bb9 >2012-12-17 08:27:01,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:01,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ef64bb9 >2012-12-17 08:27:01,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:27:01,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4544 >2012-12-17 08:27:01,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4544, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67be27d8 >2012-12-17 08:27:01,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 9267a991-579a-4158-9671-fe922c935fe5 >2012-12-17 08:27:02,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 9267a991-579a-4158-9671-fe922c935fe5 task status = finished >2012-12-17 08:27:02,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 08:27:02,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9267a991-579a-4158-9671-fe922c935fe5), log id: 63b5c46f >2012-12-17 08:27:02,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 63b5c46f >2012-12-17 08:27:02,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3273868d, log id: 67be27d8 >2012-12-17 08:27:02,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:02,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:02,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ccd6597 >2012-12-17 08:27:02,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ccd6597 >2012-12-17 08:27:02,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 38729bd7 >2012-12-17 08:27:02,804 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:02,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cb5109d >2012-12-17 08:27:02,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:02,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a93888a >2012-12-17 08:27:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a93888a >2012-12-17 08:27:02,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cb5109d >2012-12-17 08:27:02,824 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:11,685 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [3116bd71] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:11,819 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [69b76134] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:11,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 616b9f37 >2012-12-17 08:27:11,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1647fb5b >2012-12-17 08:27:11,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:12,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, SpmStopVDSCommand, log id: 1647fb5b >2012-12-17 08:27:12,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, ResetIrsVDSCommand, log id: 616b9f37 >2012-12-17 08:27:12,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1585de32 >2012-12-17 08:27:12,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:12,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1585de32 >2012-12-17 08:27:12,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:27:22,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:27:22,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4545 >2012-12-17 08:27:22,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4545, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 490c48ea >2012-12-17 08:27:22,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling started: taskId = 9ac5a3b8-2000-46af-92dd-0ec34042d29d >2012-12-17 08:27:23,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling ended: taskId = 9ac5a3b8-2000-46af-92dd-0ec34042d29d task status = finished >2012-12-17 08:27:23,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] spmStart polling ended, spm status: SPM >2012-12-17 08:27:23,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9ac5a3b8-2000-46af-92dd-0ec34042d29d), log id: 69ce8866 >2012-12-17 08:27:23,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] FINISH, HSMClearTaskVDSCommand, log id: 69ce8866 >2012-12-17 08:27:23,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [2321a59a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@765d16ce, log id: 490c48ea >2012-12-17 08:27:23,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:23,349 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:27:23,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a96fba3 >2012-12-17 08:27:23,365 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [2321a59a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:23,642 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [5b075498] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:23,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4e0d18fa >2012-12-17 08:27:23,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7eda55dc >2012-12-17 08:27:23,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:23,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, SpmStopVDSCommand, log id: 7eda55dc >2012-12-17 08:27:23,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, ResetIrsVDSCommand, log id: 4e0d18fa >2012-12-17 08:27:23,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5b075498] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b85a3a7 >2012-12-17 08:27:23,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:23,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5b075498] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b85a3a7 >2012-12-17 08:27:24,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:27:24,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4546 >2012-12-17 08:27:24,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4546, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7b722591 >2012-12-17 08:27:24,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling started: taskId = 8a1cc935-6bc1-40e6-98d3-c05ecf197826 >2012-12-17 08:27:25,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended: taskId = 8a1cc935-6bc1-40e6-98d3-c05ecf197826 task status = finished >2012-12-17 08:27:25,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended, spm status: SPM >2012-12-17 08:27:25,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a1cc935-6bc1-40e6-98d3-c05ecf197826), log id: 1b23d50c >2012-12-17 08:27:25,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMClearTaskVDSCommand, log id: 1b23d50c >2012-12-17 08:27:25,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d7814bf, log id: 7b722591 >2012-12-17 08:27:25,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:25,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:25,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f93c5ce >2012-12-17 08:27:25,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f93c5ce >2012-12-17 08:27:25,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a96fba3 >2012-12-17 08:27:25,185 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:25,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4e9f909c >2012-12-17 08:27:25,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:25,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52acef5f >2012-12-17 08:27:25,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52acef5f >2012-12-17 08:27:25,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e9f909c >2012-12-17 08:27:25,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [3b972285] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:32,831 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:27:34,059 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [7b78f9c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:34,200 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [4adae006] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:34,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 680e741 >2012-12-17 08:27:34,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62949086 >2012-12-17 08:27:34,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:34,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, SpmStopVDSCommand, log id: 62949086 >2012-12-17 08:27:34,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, ResetIrsVDSCommand, log id: 680e741 >2012-12-17 08:27:34,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44fba90c >2012-12-17 08:27:34,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:34,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44fba90c >2012-12-17 08:27:43,032 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:27:44,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:27:44,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4547 >2012-12-17 08:27:44,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4547, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fb4195d >2012-12-17 08:27:44,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling started: taskId = f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f >2012-12-17 08:27:45,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling ended: taskId = f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f task status = finished >2012-12-17 08:27:45,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] spmStart polling ended, spm status: SPM >2012-12-17 08:27:45,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7dc1354-ad82-4ee7-9f16-e0fe47cfcd3f), log id: 3469e4e6 >2012-12-17 08:27:45,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] FINISH, HSMClearTaskVDSCommand, log id: 3469e4e6 >2012-12-17 08:27:45,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [13794c1c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1432d2ec, log id: 2fb4195d >2012-12-17 08:27:45,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:46,035 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:27:46,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a105a08 >2012-12-17 08:27:46,113 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [13794c1c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:46,276 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [5b5bb621] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:46,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5698c284 >2012-12-17 08:27:46,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f0a6cd9 >2012-12-17 08:27:46,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:46,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, SpmStopVDSCommand, log id: 4f0a6cd9 >2012-12-17 08:27:46,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, ResetIrsVDSCommand, log id: 5698c284 >2012-12-17 08:27:46,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21e2bf06 >2012-12-17 08:27:46,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:46,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21e2bf06 >2012-12-17 08:27:46,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:27:46,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4548 >2012-12-17 08:27:46,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4548, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 125b03d9 >2012-12-17 08:27:46,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling started: taskId = 7ba4e369-cd91-4f1a-aa29-7c36750e15da >2012-12-17 08:27:47,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended: taskId = 7ba4e369-cd91-4f1a-aa29-7c36750e15da task status = finished >2012-12-17 08:27:47,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended, spm status: SPM >2012-12-17 08:27:47,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ba4e369-cd91-4f1a-aa29-7c36750e15da), log id: 79ca7c7 >2012-12-17 08:27:47,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMClearTaskVDSCommand, log id: 79ca7c7 >2012-12-17 08:27:47,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dcafdb8, log id: 125b03d9 >2012-12-17 08:27:47,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:27:47,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:47,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135e7da8 >2012-12-17 08:27:47,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135e7da8 >2012-12-17 08:27:47,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a105a08 >2012-12-17 08:27:47,724 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:47,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 316895c2 >2012-12-17 08:27:47,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:47,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e99b9ac >2012-12-17 08:27:47,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e99b9ac >2012-12-17 08:27:47,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 316895c2 >2012-12-17 08:27:47,744 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:27:53,217 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:27:56,610 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [24da4704] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:27:56,771 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [a209fdc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:27:56,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 473fa876 >2012-12-17 08:27:56,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 498dd5cd >2012-12-17 08:27:56,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:57,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, SpmStopVDSCommand, log id: 498dd5cd >2012-12-17 08:27:57,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, ResetIrsVDSCommand, log id: 473fa876 >2012-12-17 08:27:57,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e1fa20d >2012-12-17 08:27:57,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:27:57,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e1fa20d >2012-12-17 08:28:03,429 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:28:07,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:07,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4549 >2012-12-17 08:28:07,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4549, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12aa38ff >2012-12-17 08:28:07,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling started: taskId = f9d24c1e-3597-4134-8a1c-d3055e248267 >2012-12-17 08:28:08,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling ended: taskId = f9d24c1e-3597-4134-8a1c-d3055e248267 task status = finished >2012-12-17 08:28:08,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] spmStart polling ended, spm status: SPM >2012-12-17 08:28:08,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [43662b97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9d24c1e-3597-4134-8a1c-d3055e248267), log id: 7b40c319 >2012-12-17 08:28:08,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, HSMClearTaskVDSCommand, log id: 7b40c319 >2012-12-17 08:28:08,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [43662b97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c8dd86, log id: 12aa38ff >2012-12-17 08:28:08,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:08,304 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:28:08,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 78853209 >2012-12-17 08:28:08,325 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [43662b97] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:28:08,479 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1a2cf793] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:28:08,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1894fd64 >2012-12-17 08:28:08,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79e7068d >2012-12-17 08:28:08,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:09,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, SpmStopVDSCommand, log id: 79e7068d >2012-12-17 08:28:09,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, ResetIrsVDSCommand, log id: 1894fd64 >2012-12-17 08:28:09,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1339bb6a >2012-12-17 08:28:09,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1a2cf793] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:09,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1a2cf793] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1339bb6a >2012-12-17 08:28:09,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:09,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4550 >2012-12-17 08:28:09,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4550, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bdad4b2 >2012-12-17 08:28:09,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling started: taskId = da659046-1dcf-41d9-a9e6-3eb5a3f1e443 >2012-12-17 08:28:10,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling ended: taskId = da659046-1dcf-41d9-a9e6-3eb5a3f1e443 task status = finished >2012-12-17 08:28:10,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] spmStart polling ended, spm status: SPM >2012-12-17 08:28:10,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da659046-1dcf-41d9-a9e6-3eb5a3f1e443), log id: 6f3fbfed >2012-12-17 08:28:10,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMClearTaskVDSCommand, log id: 6f3fbfed >2012-12-17 08:28:10,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@425199aa, log id: 2bdad4b2 >2012-12-17 08:28:10,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:10,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:10,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e0f8ee5 >2012-12-17 08:28:10,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e0f8ee5 >2012-12-17 08:28:10,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78853209 >2012-12-17 08:28:10,981 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:28:10,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f6e9d0b >2012-12-17 08:28:10,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:10,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2ccacb34 >2012-12-17 08:28:11,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ccacb34 >2012-12-17 08:28:11,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [4f7e897c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f6e9d0b >2012-12-17 08:28:11,000 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [4f7e897c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:28:13,686 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:28:19,367 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [64fd8799] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:28:19,464 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [5571991a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:28:19,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1561d894 >2012-12-17 08:28:19,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f05c822 >2012-12-17 08:28:19,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:19,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, SpmStopVDSCommand, log id: 4f05c822 >2012-12-17 08:28:19,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, ResetIrsVDSCommand, log id: 1561d894 >2012-12-17 08:28:19,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [5571991a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cd73ff0 >2012-12-17 08:28:19,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:19,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [5571991a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cd73ff0 >2012-12-17 08:28:23,892 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:28:29,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:29,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4551 >2012-12-17 08:28:29,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4551, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e97d4b5 >2012-12-17 08:28:29,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling started: taskId = a0368716-d7f3-48e4-82a5-acd5be6ec3a1 >2012-12-17 08:28:30,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling ended: taskId = a0368716-d7f3-48e4-82a5-acd5be6ec3a1 task status = finished >2012-12-17 08:28:30,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] spmStart polling ended, spm status: SPM >2012-12-17 08:28:30,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a0368716-d7f3-48e4-82a5-acd5be6ec3a1), log id: fd29875 >2012-12-17 08:28:30,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] FINISH, HSMClearTaskVDSCommand, log id: fd29875 >2012-12-17 08:28:30,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [61c8d0e2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b173170, log id: e97d4b5 >2012-12-17 08:28:30,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:31,055 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:28:31,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58a0a6c0 >2012-12-17 08:28:31,066 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [61c8d0e2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:28:31,286 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [550b34b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:28:31,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e95a248 >2012-12-17 08:28:31,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 16febfc7 >2012-12-17 08:28:31,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:32,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, SpmStopVDSCommand, log id: 16febfc7 >2012-12-17 08:28:32,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, ResetIrsVDSCommand, log id: 6e95a248 >2012-12-17 08:28:32,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c5cde92 >2012-12-17 08:28:32,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [550b34b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:32,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [550b34b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c5cde92 >2012-12-17 08:28:32,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:32,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4552 >2012-12-17 08:28:32,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4552, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 141d1138 >2012-12-17 08:28:32,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = 354f971f-5700-4629-9aa7-e4c6b67757db >2012-12-17 08:28:33,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = 354f971f-5700-4629-9aa7-e4c6b67757db task status = finished >2012-12-17 08:28:33,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM >2012-12-17 08:28:33,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=354f971f-5700-4629-9aa7-e4c6b67757db), log id: f982ff8 >2012-12-17 08:28:33,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: f982ff8 >2012-12-17 08:28:33,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50754ac7, log id: 141d1138 >2012-12-17 08:28:33,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:33,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:33,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ebbb714 >2012-12-17 08:28:33,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ebbb714 >2012-12-17 08:28:33,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58a0a6c0 >2012-12-17 08:28:33,459 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:28:33,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35f091cf >2012-12-17 08:28:33,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:33,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ac5b316 >2012-12-17 08:28:33,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ac5b316 >2012-12-17 08:28:33,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35f091cf >2012-12-17 08:28:33,480 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:28:34,178 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:28:42,330 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [723e6c9b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:28:42,483 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [28319bba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:28:42,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3464f905 >2012-12-17 08:28:42,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5887b2aa >2012-12-17 08:28:42,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:43,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStopVDSCommand, log id: 5887b2aa >2012-12-17 08:28:43,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, ResetIrsVDSCommand, log id: 3464f905 >2012-12-17 08:28:43,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24c6c50d >2012-12-17 08:28:43,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:43,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24c6c50d >2012-12-17 08:28:44,373 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:28:53,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:53,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4553 >2012-12-17 08:28:53,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4553, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ca0bf09 >2012-12-17 08:28:53,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling started: taskId = f9a60457-7b64-4825-827c-c3ab45d13c0b >2012-12-17 08:28:54,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling ended: taskId = f9a60457-7b64-4825-827c-c3ab45d13c0b task status = finished >2012-12-17 08:28:54,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] spmStart polling ended, spm status: SPM >2012-12-17 08:28:54,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9a60457-7b64-4825-827c-c3ab45d13c0b), log id: 3b90741d >2012-12-17 08:28:54,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] FINISH, HSMClearTaskVDSCommand, log id: 3b90741d >2012-12-17 08:28:54,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f87b09d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8b73a78, log id: 3ca0bf09 >2012-12-17 08:28:54,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:54,701 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:28:54,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77a18334 >2012-12-17 08:28:54,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f87b09d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:28:54,824 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [780c369f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:28:54,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1268ec8f >2012-12-17 08:28:54,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6ecc85d9 >2012-12-17 08:28:54,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:55,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, SpmStopVDSCommand, log id: 6ecc85d9 >2012-12-17 08:28:55,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, ResetIrsVDSCommand, log id: 1268ec8f >2012-12-17 08:28:55,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [780c369f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 473d11d4 >2012-12-17 08:28:55,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:55,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [780c369f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 473d11d4 >2012-12-17 08:28:55,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:28:55,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4554 >2012-12-17 08:28:55,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4554, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e64bff1 >2012-12-17 08:28:55,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling started: taskId = a6036110-6bbb-4162-9520-2227b694d8bb >2012-12-17 08:28:56,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling ended: taskId = a6036110-6bbb-4162-9520-2227b694d8bb task status = finished >2012-12-17 08:28:56,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] spmStart polling ended, spm status: SPM >2012-12-17 08:28:56,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6036110-6bbb-4162-9520-2227b694d8bb), log id: 76fbcddb >2012-12-17 08:28:56,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMClearTaskVDSCommand, log id: 76fbcddb >2012-12-17 08:28:56,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@d8821e4, log id: 6e64bff1 >2012-12-17 08:28:56,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:28:56,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:56,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13fec599 >2012-12-17 08:28:56,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13fec599 >2012-12-17 08:28:56,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77a18334 >2012-12-17 08:28:56,323 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:28:56,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c23fee3 >2012-12-17 08:28:56,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:28:56,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12a62ee >2012-12-17 08:28:56,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12a62ee >2012-12-17 08:28:56,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [2b22186] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c23fee3 >2012-12-17 08:28:56,343 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [2b22186] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:29:04,777 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:29:05,217 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [76a14bbb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:29:06,616 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [1981e345] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:29:06,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 121a7acd >2012-12-17 08:29:06,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2074534 >2012-12-17 08:29:06,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:06,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SpmStopVDSCommand, log id: 2074534 >2012-12-17 08:29:06,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, ResetIrsVDSCommand, log id: 121a7acd >2012-12-17 08:29:06,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ad93aa1 >2012-12-17 08:29:06,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:06,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ad93aa1 >2012-12-17 08:29:14,962 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:29:17,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:29:17,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4555 >2012-12-17 08:29:17,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4555, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55af6880 >2012-12-17 08:29:17,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling started: taskId = ae0410fe-65fd-47e8-a52a-2ff86a2e7e13 >2012-12-17 08:29:18,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling ended: taskId = ae0410fe-65fd-47e8-a52a-2ff86a2e7e13 task status = finished >2012-12-17 08:29:18,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] spmStart polling ended, spm status: SPM >2012-12-17 08:29:18,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [6462735f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae0410fe-65fd-47e8-a52a-2ff86a2e7e13), log id: 6ef9b666 >2012-12-17 08:29:18,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, HSMClearTaskVDSCommand, log id: 6ef9b666 >2012-12-17 08:29:18,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [6462735f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c361b75, log id: 55af6880 >2012-12-17 08:29:18,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:29:18,296 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:29:18,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 114c1056 >2012-12-17 08:29:18,308 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [6462735f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:29:18,506 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [65048227] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:29:18,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20390a05 >2012-12-17 08:29:18,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e4449d9 >2012-12-17 08:29:18,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:18,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStopVDSCommand, log id: e4449d9 >2012-12-17 08:29:18,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, ResetIrsVDSCommand, log id: 20390a05 >2012-12-17 08:29:18,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 556f1d5f >2012-12-17 08:29:18,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:18,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 556f1d5f >2012-12-17 08:29:18,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:29:18,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4556 >2012-12-17 08:29:18,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4556, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f4c6985 >2012-12-17 08:29:18,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling started: taskId = 8d2c8abb-150d-48c6-b08a-6ef288aef0b0 >2012-12-17 08:29:19,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended: taskId = 8d2c8abb-150d-48c6-b08a-6ef288aef0b0 task status = finished >2012-12-17 08:29:19,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended, spm status: SPM >2012-12-17 08:29:19,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8d2c8abb-150d-48c6-b08a-6ef288aef0b0), log id: 6533f080 >2012-12-17 08:29:19,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMClearTaskVDSCommand, log id: 6533f080 >2012-12-17 08:29:19,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34f29768, log id: f4c6985 >2012-12-17 08:29:20,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:29:20,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:20,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 149d45f6 >2012-12-17 08:29:20,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 149d45f6 >2012-12-17 08:29:20,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 114c1056 >2012-12-17 08:29:20,058 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:29:20,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 615c240c >2012-12-17 08:29:20,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:20,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4447bffd >2012-12-17 08:29:20,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4447bffd >2012-12-17 08:29:20,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 615c240c >2012-12-17 08:29:20,078 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [54cb0ab7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:29:26,208 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:29:28,861 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3bf47b4b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:29:28,998 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [73dddeb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:29:29,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40c9897d >2012-12-17 08:29:29,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fa569ea >2012-12-17 08:29:29,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:29,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, SpmStopVDSCommand, log id: 2fa569ea >2012-12-17 08:29:29,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, ResetIrsVDSCommand, log id: 40c9897d >2012-12-17 08:29:29,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 298898f8 >2012-12-17 08:29:29,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:29,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 298898f8 >2012-12-17 08:29:36,637 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:29:39,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:29:39,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4557 >2012-12-17 08:29:39,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4557, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 41fd88e1 >2012-12-17 08:29:39,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling started: taskId = fb61d30d-53a2-4a50-8f7a-58bd721059d0 >2012-12-17 08:29:40,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling ended: taskId = fb61d30d-53a2-4a50-8f7a-58bd721059d0 task status = finished >2012-12-17 08:29:40,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] spmStart polling ended, spm status: SPM >2012-12-17 08:29:40,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fb61d30d-53a2-4a50-8f7a-58bd721059d0), log id: 4bf777d0 >2012-12-17 08:29:40,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] FINISH, HSMClearTaskVDSCommand, log id: 4bf777d0 >2012-12-17 08:29:40,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [17cf9677] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63db1fa6, log id: 41fd88e1 >2012-12-17 08:29:41,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:29:41,726 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:29:41,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69267410 >2012-12-17 08:29:41,763 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [17cf9677] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:29:41,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [57c34106] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:29:42,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1c428764 >2012-12-17 08:29:42,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dccff29 >2012-12-17 08:29:42,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:42,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, SpmStopVDSCommand, log id: 5dccff29 >2012-12-17 08:29:42,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, ResetIrsVDSCommand, log id: 1c428764 >2012-12-17 08:29:42,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57c34106] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ea000ed >2012-12-17 08:29:42,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:42,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57c34106] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ea000ed >2012-12-17 08:29:42,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:29:42,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4558 >2012-12-17 08:29:42,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4558, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 779b4d44 >2012-12-17 08:29:42,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling started: taskId = 217dcb1f-231d-4eed-931a-a9e66ca68ab7 >2012-12-17 08:29:43,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended: taskId = 217dcb1f-231d-4eed-931a-a9e66ca68ab7 task status = finished >2012-12-17 08:29:43,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended, spm status: SPM >2012-12-17 08:29:43,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=217dcb1f-231d-4eed-931a-a9e66ca68ab7), log id: 348435c0 >2012-12-17 08:29:43,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMClearTaskVDSCommand, log id: 348435c0 >2012-12-17 08:29:43,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53e4eb2f, log id: 779b4d44 >2012-12-17 08:29:43,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:29:43,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:43,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2491ab1c >2012-12-17 08:29:43,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2491ab1c >2012-12-17 08:29:43,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69267410 >2012-12-17 08:29:43,512 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:29:43,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c1dd720 >2012-12-17 08:29:43,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:43,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 786b1035 >2012-12-17 08:29:43,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 786b1035 >2012-12-17 08:29:43,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c1dd720 >2012-12-17 08:29:43,531 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [51b75138] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:29:46,814 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:29:52,337 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [48110f6e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:29:52,448 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [369a76fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:29:52,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ddb57b >2012-12-17 08:29:52,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: afc1624 >2012-12-17 08:29:52,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:52,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, SpmStopVDSCommand, log id: afc1624 >2012-12-17 08:29:52,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, ResetIrsVDSCommand, log id: 13ddb57b >2012-12-17 08:29:52,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76a97685 >2012-12-17 08:29:52,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:29:52,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76a97685 >2012-12-17 08:29:57,028 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Checking autorecoverable hosts >2012-12-17 08:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Autorecovering 0 hosts >2012-12-17 08:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Checking autorecoverable hosts done >2012-12-17 08:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [664081c5] Autorecovering storage domains is disabled, skipping >2012-12-17 08:30:02,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:02,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4559 >2012-12-17 08:30:02,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4559, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ac2fbbc >2012-12-17 08:30:02,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling started: taskId = 217573cb-1322-4fe6-bf3b-e2a426484469 >2012-12-17 08:30:03,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling ended: taskId = 217573cb-1322-4fe6-bf3b-e2a426484469 task status = finished >2012-12-17 08:30:03,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] spmStart polling ended, spm status: SPM >2012-12-17 08:30:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [376f3052] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=217573cb-1322-4fe6-bf3b-e2a426484469), log id: f69603d >2012-12-17 08:30:03,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, HSMClearTaskVDSCommand, log id: f69603d >2012-12-17 08:30:03,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [376f3052] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f2d4cb2, log id: 1ac2fbbc >2012-12-17 08:30:03,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:03,917 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:30:03,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 223c183 >2012-12-17 08:30:03,943 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [376f3052] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:30:04,097 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [b1109c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:30:04,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13b1f5ce >2012-12-17 08:30:04,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d7d70b6 >2012-12-17 08:30:04,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:04,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, SpmStopVDSCommand, log id: 3d7d70b6 >2012-12-17 08:30:04,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, ResetIrsVDSCommand, log id: 13b1f5ce >2012-12-17 08:30:04,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e737678 >2012-12-17 08:30:04,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [b1109c1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:04,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [b1109c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e737678 >2012-12-17 08:30:04,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:04,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4560 >2012-12-17 08:30:04,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4560, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69f10d48 >2012-12-17 08:30:04,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling started: taskId = a222a6c2-ab4f-49f8-8a64-8339516e2999 >2012-12-17 08:30:05,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended: taskId = a222a6c2-ab4f-49f8-8a64-8339516e2999 task status = finished >2012-12-17 08:30:05,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended, spm status: SPM >2012-12-17 08:30:05,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a222a6c2-ab4f-49f8-8a64-8339516e2999), log id: 13051223 >2012-12-17 08:30:05,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMClearTaskVDSCommand, log id: 13051223 >2012-12-17 08:30:05,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31e9c196, log id: 69f10d48 >2012-12-17 08:30:06,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:06,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:06,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42062e4f >2012-12-17 08:30:06,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42062e4f >2012-12-17 08:30:06,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 223c183 >2012-12-17 08:30:06,270 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:06,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d08c4e5 >2012-12-17 08:30:06,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:06,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d4004b5 >2012-12-17 08:30:06,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d4004b5 >2012-12-17 08:30:06,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d08c4e5 >2012-12-17 08:30:06,291 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:07,238 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:30:14,744 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [6bdbfd31] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:30:14,841 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [4557a90c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:30:14,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12c594e9 >2012-12-17 08:30:14,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68bcf9f3 >2012-12-17 08:30:14,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:15,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, SpmStopVDSCommand, log id: 68bcf9f3 >2012-12-17 08:30:15,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, ResetIrsVDSCommand, log id: 12c594e9 >2012-12-17 08:30:15,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45a97ba2 >2012-12-17 08:30:15,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:15,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [4557a90c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45a97ba2 >2012-12-17 08:30:17,483 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:30:25,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:25,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4561 >2012-12-17 08:30:25,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4561, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15f6249b >2012-12-17 08:30:25,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling started: taskId = 82afa13a-e7da-4946-ad42-eba6c560549e >2012-12-17 08:30:26,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling ended: taskId = 82afa13a-e7da-4946-ad42-eba6c560549e task status = finished >2012-12-17 08:30:26,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] spmStart polling ended, spm status: SPM >2012-12-17 08:30:26,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [152503f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=82afa13a-e7da-4946-ad42-eba6c560549e), log id: 235c5b75 >2012-12-17 08:30:26,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, HSMClearTaskVDSCommand, log id: 235c5b75 >2012-12-17 08:30:26,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [152503f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4787076, log id: 15f6249b >2012-12-17 08:30:26,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:26,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:30:26,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 114bf81 >2012-12-17 08:30:26,540 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [152503f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:30:26,634 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [733172b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:30:26,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6832647e >2012-12-17 08:30:26,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ab668e1 >2012-12-17 08:30:26,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:26,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStopVDSCommand, log id: ab668e1 >2012-12-17 08:30:26,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, ResetIrsVDSCommand, log id: 6832647e >2012-12-17 08:30:26,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2221f964 >2012-12-17 08:30:26,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:26,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2221f964 >2012-12-17 08:30:26,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:26,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4562 >2012-12-17 08:30:26,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4562, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b561214 >2012-12-17 08:30:27,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling started: taskId = 453e98c1-8acb-4fb8-bdbc-8838936643e3 >2012-12-17 08:30:28,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling ended: taskId = 453e98c1-8acb-4fb8-bdbc-8838936643e3 task status = finished >2012-12-17 08:30:28,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] spmStart polling ended, spm status: SPM >2012-12-17 08:30:28,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=453e98c1-8acb-4fb8-bdbc-8838936643e3), log id: b9d0271 >2012-12-17 08:30:28,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMClearTaskVDSCommand, log id: b9d0271 >2012-12-17 08:30:28,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@710b4eab, log id: b561214 >2012-12-17 08:30:28,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:28,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:28,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4911fe04 >2012-12-17 08:30:28,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4911fe04 >2012-12-17 08:30:28,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 114bf81 >2012-12-17 08:30:28,146 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:28,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e71953e >2012-12-17 08:30:28,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:28,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 64f70287 >2012-12-17 08:30:28,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 64f70287 >2012-12-17 08:30:28,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [16c2b2c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e71953e >2012-12-17 08:30:28,166 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [16c2b2c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:37,051 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [55210d2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:30:37,176 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [169ff372] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:30:37,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8033687 >2012-12-17 08:30:37,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: bdcb13e >2012-12-17 08:30:37,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:37,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStopVDSCommand, log id: bdcb13e >2012-12-17 08:30:37,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, ResetIrsVDSCommand, log id: 8033687 >2012-12-17 08:30:37,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 534af002 >2012-12-17 08:30:37,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:37,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 534af002 >2012-12-17 08:30:37,879 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:30:47,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:47,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4563 >2012-12-17 08:30:47,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4563, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1be93b3e >2012-12-17 08:30:47,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling started: taskId = bb692280-bb1b-46cc-a7bc-9f39f57063bf >2012-12-17 08:30:48,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling ended: taskId = bb692280-bb1b-46cc-a7bc-9f39f57063bf task status = finished >2012-12-17 08:30:48,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] spmStart polling ended, spm status: SPM >2012-12-17 08:30:48,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb692280-bb1b-46cc-a7bc-9f39f57063bf), log id: 53b84286 >2012-12-17 08:30:48,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, HSMClearTaskVDSCommand, log id: 53b84286 >2012-12-17 08:30:48,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [7bc37769] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6617965a, log id: 1be93b3e >2012-12-17 08:30:48,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:48,689 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:30:48,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 609d6f9a >2012-12-17 08:30:48,721 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [7bc37769] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:30:48,893 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [181d86f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:30:48,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 55c26eba >2012-12-17 08:30:48,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1554053b >2012-12-17 08:30:49,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:49,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStopVDSCommand, log id: 1554053b >2012-12-17 08:30:49,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, ResetIrsVDSCommand, log id: 55c26eba >2012-12-17 08:30:49,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d729433 >2012-12-17 08:30:49,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:49,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d729433 >2012-12-17 08:30:49,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:30:50,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4564 >2012-12-17 08:30:50,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4564, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d29fbd8 >2012-12-17 08:30:50,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling started: taskId = 1ed39c45-91ff-4337-82c2-75af88e9146c >2012-12-17 08:30:51,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended: taskId = 1ed39c45-91ff-4337-82c2-75af88e9146c task status = finished >2012-12-17 08:30:51,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended, spm status: SPM >2012-12-17 08:30:51,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1ed39c45-91ff-4337-82c2-75af88e9146c), log id: 66646150 >2012-12-17 08:30:51,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMClearTaskVDSCommand, log id: 66646150 >2012-12-17 08:30:51,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4830ffb4, log id: 4d29fbd8 >2012-12-17 08:30:51,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:30:51,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:51,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 17ab2de8 >2012-12-17 08:30:51,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 17ab2de8 >2012-12-17 08:30:51,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 609d6f9a >2012-12-17 08:30:51,594 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:51,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7399ca3b >2012-12-17 08:30:51,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:30:51,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 110e1121 >2012-12-17 08:30:51,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 110e1121 >2012-12-17 08:30:51,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7399ca3b >2012-12-17 08:30:51,614 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:30:58,233 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [35e0a925] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:30:59,986 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [57b06faf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:00,257 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [69e82ff1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:01,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 122a3a39 >2012-12-17 08:31:01,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 43e23c4b >2012-12-17 08:31:01,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:01,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, SpmStopVDSCommand, log id: 43e23c4b >2012-12-17 08:31:01,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, ResetIrsVDSCommand, log id: 122a3a39 >2012-12-17 08:31:01,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bda5372 >2012-12-17 08:31:01,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:01,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bda5372 >2012-12-17 08:31:08,472 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:31:11,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:31:11,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4565 >2012-12-17 08:31:11,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4565, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30ca9c00 >2012-12-17 08:31:11,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling started: taskId = 2e43d571-fb34-4d9a-9d9f-b7b60ea6256d >2012-12-17 08:31:12,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling ended: taskId = 2e43d571-fb34-4d9a-9d9f-b7b60ea6256d task status = finished >2012-12-17 08:31:12,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] spmStart polling ended, spm status: SPM >2012-12-17 08:31:12,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e43d571-fb34-4d9a-9d9f-b7b60ea6256d), log id: 22a84930 >2012-12-17 08:31:12,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, HSMClearTaskVDSCommand, log id: 22a84930 >2012-12-17 08:31:12,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [53e8b92c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@565d0016, log id: 30ca9c00 >2012-12-17 08:31:12,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:31:12,693 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:31:12,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 24a70eff >2012-12-17 08:31:12,712 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [53e8b92c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:12,899 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [57e0db40] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:12,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: bc8e6d2 >2012-12-17 08:31:12,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d03bc1d >2012-12-17 08:31:12,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:13,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, SpmStopVDSCommand, log id: 2d03bc1d >2012-12-17 08:31:13,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, ResetIrsVDSCommand, log id: bc8e6d2 >2012-12-17 08:31:13,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bf14db5 >2012-12-17 08:31:13,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:13,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bf14db5 >2012-12-17 08:31:13,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:31:13,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4566 >2012-12-17 08:31:13,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4566, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16d960f >2012-12-17 08:31:13,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling started: taskId = be25005c-041d-45bf-90cd-11bad888cc93 >2012-12-17 08:31:14,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling ended: taskId = be25005c-041d-45bf-90cd-11bad888cc93 task status = finished >2012-12-17 08:31:14,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] spmStart polling ended, spm status: SPM >2012-12-17 08:31:14,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=be25005c-041d-45bf-90cd-11bad888cc93), log id: 6c3457f1 >2012-12-17 08:31:14,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMClearTaskVDSCommand, log id: 6c3457f1 >2012-12-17 08:31:14,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6079c6b6, log id: 16d960f >2012-12-17 08:31:14,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:31:14,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:14,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6457e629 >2012-12-17 08:31:14,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6457e629 >2012-12-17 08:31:14,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 24a70eff >2012-12-17 08:31:14,887 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:31:14,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46c7c136 >2012-12-17 08:31:14,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:14,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 655a6acb >2012-12-17 08:31:14,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 655a6acb >2012-12-17 08:31:14,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [65976fd0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46c7c136 >2012-12-17 08:31:14,908 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [65976fd0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:31:18,742 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:31:23,695 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [311c1434] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:23,873 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [f91baee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:23,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2909d058 >2012-12-17 08:31:23,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76220ced >2012-12-17 08:31:23,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:24,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, SpmStopVDSCommand, log id: 76220ced >2012-12-17 08:31:24,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, ResetIrsVDSCommand, log id: 2909d058 >2012-12-17 08:31:24,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3ceddd9b >2012-12-17 08:31:24,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:24,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3ceddd9b >2012-12-17 08:31:29,190 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:31:34,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:31:34,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4567 >2012-12-17 08:31:34,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4567, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35a2d58e >2012-12-17 08:31:34,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling started: taskId = 28958245-1f85-4593-99f3-adf6374a61af >2012-12-17 08:31:35,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling ended: taskId = 28958245-1f85-4593-99f3-adf6374a61af task status = finished >2012-12-17 08:31:35,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] spmStart polling ended, spm status: SPM >2012-12-17 08:31:35,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=28958245-1f85-4593-99f3-adf6374a61af), log id: 3a087a18 >2012-12-17 08:31:35,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] FINISH, HSMClearTaskVDSCommand, log id: 3a087a18 >2012-12-17 08:31:35,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1a53ba46] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@287db6c, log id: 35a2d58e >2012-12-17 08:31:35,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:31:36,414 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:31:36,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 606f7d3c >2012-12-17 08:31:36,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1a53ba46] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:36,623 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [27ff9aa4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:36,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 458c9c83 >2012-12-17 08:31:36,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 216e500c >2012-12-17 08:31:36,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:36,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, SpmStopVDSCommand, log id: 216e500c >2012-12-17 08:31:36,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, ResetIrsVDSCommand, log id: 458c9c83 >2012-12-17 08:31:36,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b8eb76d >2012-12-17 08:31:36,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [27ff9aa4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:36,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [27ff9aa4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b8eb76d >2012-12-17 08:31:36,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:31:36,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4568 >2012-12-17 08:31:36,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4568, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ceff4fa >2012-12-17 08:31:36,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling started: taskId = d72d3eec-fac8-4bba-8176-14d19152d16e >2012-12-17 08:31:38,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended: taskId = d72d3eec-fac8-4bba-8176-14d19152d16e task status = finished >2012-12-17 08:31:38,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended, spm status: SPM >2012-12-17 08:31:38,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d72d3eec-fac8-4bba-8176-14d19152d16e), log id: 67aeda9e >2012-12-17 08:31:38,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMClearTaskVDSCommand, log id: 67aeda9e >2012-12-17 08:31:38,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@892f040, log id: ceff4fa >2012-12-17 08:31:38,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:31:38,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:38,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4f5ee3b9 >2012-12-17 08:31:38,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4f5ee3b9 >2012-12-17 08:31:38,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 606f7d3c >2012-12-17 08:31:38,125 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:31:38,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5eb8af64 >2012-12-17 08:31:38,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:38,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 66b9a70c >2012-12-17 08:31:38,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66b9a70c >2012-12-17 08:31:38,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5eb8af64 >2012-12-17 08:31:38,144 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [2e21b599] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:31:39,409 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:31:46,974 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [35e0a925] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:47,104 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [2edbd860] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:47,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 768b098a >2012-12-17 08:31:47,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6defbb08 >2012-12-17 08:31:47,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:47,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SpmStopVDSCommand, log id: 6defbb08 >2012-12-17 08:31:47,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, ResetIrsVDSCommand, log id: 768b098a >2012-12-17 08:31:47,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78c751c8 >2012-12-17 08:31:47,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:31:47,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78c751c8 >2012-12-17 08:31:49,763 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:31:58,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:31:58,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4569 >2012-12-17 08:31:58,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4569, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7b61ee0a >2012-12-17 08:31:58,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling started: taskId = d3593a90-81f4-49a8-a78e-b153569d87db >2012-12-17 08:31:59,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling ended: taskId = d3593a90-81f4-49a8-a78e-b153569d87db task status = finished >2012-12-17 08:31:59,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] spmStart polling ended, spm status: SPM >2012-12-17 08:31:59,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [69b76134] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d3593a90-81f4-49a8-a78e-b153569d87db), log id: 61bc802e >2012-12-17 08:31:59,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, HSMClearTaskVDSCommand, log id: 61bc802e >2012-12-17 08:31:59,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [69b76134] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e837e23, log id: 7b61ee0a >2012-12-17 08:31:59,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:31:59,253 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:31:59,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3929ad27 >2012-12-17 08:31:59,272 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [69b76134] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:31:59,406 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [6f019f85] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:31:59,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e59089d >2012-12-17 08:31:59,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b3af3f9 >2012-12-17 08:31:59,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:00,008 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:32:00,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, SpmStopVDSCommand, log id: 7b3af3f9 >2012-12-17 08:32:01,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, ResetIrsVDSCommand, log id: 5e59089d >2012-12-17 08:32:01,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fd3326c >2012-12-17 08:32:01,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:01,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fd3326c >2012-12-17 08:32:01,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:32:01,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4570 >2012-12-17 08:32:01,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4570, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 700fcf5e >2012-12-17 08:32:01,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = e04167e8-338a-4376-a30b-b01cd102e874 >2012-12-17 08:32:02,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = e04167e8-338a-4376-a30b-b01cd102e874 task status = finished >2012-12-17 08:32:02,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM >2012-12-17 08:32:02,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e04167e8-338a-4376-a30b-b01cd102e874), log id: 6bb7ab1f >2012-12-17 08:32:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 6bb7ab1f >2012-12-17 08:32:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6743b778, log id: 700fcf5e >2012-12-17 08:32:02,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:32:02,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:02,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74c7161b >2012-12-17 08:32:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74c7161b >2012-12-17 08:32:02,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3929ad27 >2012-12-17 08:32:02,482 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:02,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a7c8282 >2012-12-17 08:32:02,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:02,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c4307ea >2012-12-17 08:32:02,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c4307ea >2012-12-17 08:32:02,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a7c8282 >2012-12-17 08:32:02,502 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:02,521 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:32:02,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a1e5dfd >2012-12-17 08:32:02,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:02,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e6e0e5e >2012-12-17 08:32:02,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e6e0e5e >2012-12-17 08:32:02,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a1e5dfd >2012-12-17 08:32:02,541 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:02,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b715d57 >2012-12-17 08:32:02,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:02,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 379a21f4 >2012-12-17 08:32:02,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 379a21f4 >2012-12-17 08:32:02,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b715d57 >2012-12-17 08:32:02,561 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:11,386 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1eda0292] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:32:11,662 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [70373480] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:32:11,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54c8952c >2012-12-17 08:32:11,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d546d52 >2012-12-17 08:32:11,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:12,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, SpmStopVDSCommand, log id: 6d546d52 >2012-12-17 08:32:12,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, ResetIrsVDSCommand, log id: 54c8952c >2012-12-17 08:32:12,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5466df99 >2012-12-17 08:32:12,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:12,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5466df99 >2012-12-17 08:32:21,008 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:32:22,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:32:22,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4571 >2012-12-17 08:32:22,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4571, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4dd03bec >2012-12-17 08:32:22,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling started: taskId = ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc >2012-12-17 08:32:23,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling ended: taskId = ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc task status = finished >2012-12-17 08:32:23,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] spmStart polling ended, spm status: SPM >2012-12-17 08:32:23,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [664081c5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ddfd3ff5-6724-43e7-bfe0-fb09f4a920dc), log id: 47755e2a >2012-12-17 08:32:23,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, HSMClearTaskVDSCommand, log id: 47755e2a >2012-12-17 08:32:23,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [664081c5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a793c0c, log id: 4dd03bec >2012-12-17 08:32:23,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:32:23,507 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:32:23,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5baf5588 >2012-12-17 08:32:23,547 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [664081c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:32:23,704 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:32:23,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23916919 >2012-12-17 08:32:23,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57791c21 >2012-12-17 08:32:23,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:23,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SpmStopVDSCommand, log id: 57791c21 >2012-12-17 08:32:23,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, ResetIrsVDSCommand, log id: 23916919 >2012-12-17 08:32:24,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cc4080d >2012-12-17 08:32:24,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:24,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cc4080d >2012-12-17 08:32:24,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:32:24,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4572 >2012-12-17 08:32:24,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4572, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b50e055 >2012-12-17 08:32:24,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 7863d8ea-ba19-4738-8d6d-eaf254a1d9ed >2012-12-17 08:32:25,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 7863d8ea-ba19-4738-8d6d-eaf254a1d9ed task status = finished >2012-12-17 08:32:25,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM >2012-12-17 08:32:25,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7863d8ea-ba19-4738-8d6d-eaf254a1d9ed), log id: 2b948005 >2012-12-17 08:32:25,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 2b948005 >2012-12-17 08:32:25,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2260de9d, log id: 2b50e055 >2012-12-17 08:32:25,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:32:25,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:25,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7c07758e >2012-12-17 08:32:25,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7c07758e >2012-12-17 08:32:25,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5baf5588 >2012-12-17 08:32:25,682 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:25,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4db6fd3f >2012-12-17 08:32:25,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:25,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f1bb006 >2012-12-17 08:32:25,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f1bb006 >2012-12-17 08:32:25,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4db6fd3f >2012-12-17 08:32:25,702 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:31,301 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:32:34,217 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3ff768c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:32:34,324 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:32:34,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a8b02d8 >2012-12-17 08:32:34,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5766d89a >2012-12-17 08:32:34,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:35,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, SpmStopVDSCommand, log id: 5766d89a >2012-12-17 08:32:35,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, ResetIrsVDSCommand, log id: 2a8b02d8 >2012-12-17 08:32:35,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b4180b >2012-12-17 08:32:35,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:35,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b4180b >2012-12-17 08:32:41,809 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:32:45,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:32:45,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4573 >2012-12-17 08:32:45,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4573, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5609ce79 >2012-12-17 08:32:45,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling started: taskId = 649c9b80-83d5-4286-b69e-979de9adb8f5 >2012-12-17 08:32:46,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling ended: taskId = 649c9b80-83d5-4286-b69e-979de9adb8f5 task status = finished >2012-12-17 08:32:46,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] spmStart polling ended, spm status: SPM >2012-12-17 08:32:46,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=649c9b80-83d5-4286-b69e-979de9adb8f5), log id: 1e674106 >2012-12-17 08:32:46,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] FINISH, HSMClearTaskVDSCommand, log id: 1e674106 >2012-12-17 08:32:46,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [f50ed1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@cde0082, log id: 5609ce79 >2012-12-17 08:32:46,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:32:46,770 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:32:46,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62f8dfcc >2012-12-17 08:32:46,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [f50ed1e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:32:46,977 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:32:47,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 56430a0f >2012-12-17 08:32:47,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e2dea80 >2012-12-17 08:32:47,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:47,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, SpmStopVDSCommand, log id: 6e2dea80 >2012-12-17 08:32:47,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, ResetIrsVDSCommand, log id: 56430a0f >2012-12-17 08:32:47,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bee922 >2012-12-17 08:32:47,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:47,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [48f6cd0b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bee922 >2012-12-17 08:32:47,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:32:47,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4574 >2012-12-17 08:32:47,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4574, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3169d127 >2012-12-17 08:32:47,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling started: taskId = dfa6a6ef-692b-4117-8e15-af5eec87f06d >2012-12-17 08:32:48,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling ended: taskId = dfa6a6ef-692b-4117-8e15-af5eec87f06d task status = finished >2012-12-17 08:32:48,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] spmStart polling ended, spm status: SPM >2012-12-17 08:32:48,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dfa6a6ef-692b-4117-8e15-af5eec87f06d), log id: 10978e8c >2012-12-17 08:32:48,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMClearTaskVDSCommand, log id: 10978e8c >2012-12-17 08:32:48,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e0f91d6, log id: 3169d127 >2012-12-17 08:32:48,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:32:48,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:48,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c19855f >2012-12-17 08:32:48,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c19855f >2012-12-17 08:32:48,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 62f8dfcc >2012-12-17 08:32:48,558 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:48,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f833697 >2012-12-17 08:32:48,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:48,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3471569e >2012-12-17 08:32:48,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3471569e >2012-12-17 08:32:48,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [495d106] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f833697 >2012-12-17 08:32:48,578 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [495d106] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:32:52,088 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:32:57,433 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [4f7e897c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:32:57,698 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [7b0d0943] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:32:57,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: d0d6178 >2012-12-17 08:32:57,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 31ff602 >2012-12-17 08:32:57,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:58,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStopVDSCommand, log id: 31ff602 >2012-12-17 08:32:58,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, ResetIrsVDSCommand, log id: d0d6178 >2012-12-17 08:32:58,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2892c54f >2012-12-17 08:32:58,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:32:58,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2892c54f >2012-12-17 08:33:02,301 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:33:08,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:08,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4575 >2012-12-17 08:33:08,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4575, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5812546 >2012-12-17 08:33:08,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling started: taskId = 7e1bc965-cae2-4feb-8dc5-1ba5b90a996f >2012-12-17 08:33:09,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling ended: taskId = 7e1bc965-cae2-4feb-8dc5-1ba5b90a996f task status = finished >2012-12-17 08:33:09,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] spmStart polling ended, spm status: SPM >2012-12-17 08:33:09,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [4adae006] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e1bc965-cae2-4feb-8dc5-1ba5b90a996f), log id: 76299571 >2012-12-17 08:33:09,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, HSMClearTaskVDSCommand, log id: 76299571 >2012-12-17 08:33:09,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [4adae006] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24126a52, log id: 5812546 >2012-12-17 08:33:09,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:09,799 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:33:09,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7785b9b8 >2012-12-17 08:33:09,819 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [4adae006] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:33:09,941 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:33:10,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e5a71a4 >2012-12-17 08:33:10,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69ef467e >2012-12-17 08:33:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:10,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, SpmStopVDSCommand, log id: 69ef467e >2012-12-17 08:33:11,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, ResetIrsVDSCommand, log id: 5e5a71a4 >2012-12-17 08:33:11,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f56ec7e >2012-12-17 08:33:11,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:11,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f56ec7e >2012-12-17 08:33:11,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:11,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4576 >2012-12-17 08:33:11,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4576, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c1bdff2 >2012-12-17 08:33:11,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling started: taskId = aa93f088-586b-46a2-8a46-2871b06435c3 >2012-12-17 08:33:12,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended: taskId = aa93f088-586b-46a2-8a46-2871b06435c3 task status = finished >2012-12-17 08:33:12,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended, spm status: SPM >2012-12-17 08:33:12,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aa93f088-586b-46a2-8a46-2871b06435c3), log id: 3377cf24 >2012-12-17 08:33:12,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMClearTaskVDSCommand, log id: 3377cf24 >2012-12-17 08:33:12,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ec05ee1, log id: 3c1bdff2 >2012-12-17 08:33:12,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:12,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:12,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: bcaa21a >2012-12-17 08:33:12,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: bcaa21a >2012-12-17 08:33:12,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7785b9b8 >2012-12-17 08:33:12,478 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:33:12,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 24585f02 >2012-12-17 08:33:12,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:12,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ff3349c >2012-12-17 08:33:12,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ff3349c >2012-12-17 08:33:12,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 24585f02 >2012-12-17 08:33:12,521 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:33:12,621 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:33:21,461 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [58c4b5b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:33:21,831 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [46701658] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:33:21,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 74fb834a >2012-12-17 08:33:21,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57a4de45 >2012-12-17 08:33:21,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:22,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStopVDSCommand, log id: 57a4de45 >2012-12-17 08:33:22,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, ResetIrsVDSCommand, log id: 74fb834a >2012-12-17 08:33:22,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bcbe5fb >2012-12-17 08:33:22,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:22,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bcbe5fb >2012-12-17 08:33:22,889 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:33:32,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:32,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4577 >2012-12-17 08:33:32,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4577, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28e3af35 >2012-12-17 08:33:32,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling started: taskId = 1dc1d66c-7eee-422d-8a30-6246a1a9de9f >2012-12-17 08:33:33,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling ended: taskId = 1dc1d66c-7eee-422d-8a30-6246a1a9de9f task status = finished >2012-12-17 08:33:33,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] spmStart polling ended, spm status: SPM >2012-12-17 08:33:33,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dc1d66c-7eee-422d-8a30-6246a1a9de9f), log id: 5281e7ec >2012-12-17 08:33:33,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, HSMClearTaskVDSCommand, log id: 5281e7ec >2012-12-17 08:33:33,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [a209fdc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ea16145, log id: 28e3af35 >2012-12-17 08:33:33,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:33,572 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:33:33,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cb16a82 >2012-12-17 08:33:33,606 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [a209fdc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:33:33,797 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [47f0782e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:33:33,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1743c19e >2012-12-17 08:33:33,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 369be0bb >2012-12-17 08:33:33,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:34,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, SpmStopVDSCommand, log id: 369be0bb >2012-12-17 08:33:34,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, ResetIrsVDSCommand, log id: 1743c19e >2012-12-17 08:33:34,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49ac0c7f >2012-12-17 08:33:34,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [47f0782e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:34,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [47f0782e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49ac0c7f >2012-12-17 08:33:34,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:34,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4578 >2012-12-17 08:33:34,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4578, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18caed9f >2012-12-17 08:33:34,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling started: taskId = 9fc2c21d-14e5-4410-875e-a6a49d4d27d1 >2012-12-17 08:33:35,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended: taskId = 9fc2c21d-14e5-4410-875e-a6a49d4d27d1 task status = finished >2012-12-17 08:33:35,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] spmStart polling ended, spm status: SPM >2012-12-17 08:33:35,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9fc2c21d-14e5-4410-875e-a6a49d4d27d1), log id: 188a4b74 >2012-12-17 08:33:35,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMClearTaskVDSCommand, log id: 188a4b74 >2012-12-17 08:33:35,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f5fb942, log id: 18caed9f >2012-12-17 08:33:35,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:36,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:36,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9bb4fb1 >2012-12-17 08:33:36,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9bb4fb1 >2012-12-17 08:33:36,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cb16a82 >2012-12-17 08:33:36,165 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:33:36,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 686cf274 >2012-12-17 08:33:36,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:36,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51299f41 >2012-12-17 08:33:36,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51299f41 >2012-12-17 08:33:36,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [676a97fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 686cf274 >2012-12-17 08:33:36,184 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [676a97fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:33:43,521 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:33:44,563 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5b075498] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:33:44,675 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [95f5063] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:33:44,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d3c3e15 >2012-12-17 08:33:44,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6ad320c7 >2012-12-17 08:33:44,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:44,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SpmStopVDSCommand, log id: 6ad320c7 >2012-12-17 08:33:44,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, ResetIrsVDSCommand, log id: 1d3c3e15 >2012-12-17 08:33:44,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d973f94 >2012-12-17 08:33:44,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:44,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d973f94 >2012-12-17 08:33:53,846 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:33:54,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:55,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4579 >2012-12-17 08:33:55,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4579, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 644da0c1 >2012-12-17 08:33:55,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling started: taskId = b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec >2012-12-17 08:33:56,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling ended: taskId = b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec task status = finished >2012-12-17 08:33:56,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] spmStart polling ended, spm status: SPM >2012-12-17 08:33:56,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1e20c66-2f90-43fa-8ef2-b58a3b9a6fec), log id: 4673fb46 >2012-12-17 08:33:56,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, HSMClearTaskVDSCommand, log id: 4673fb46 >2012-12-17 08:33:56,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [36fcbeec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6875d4cc, log id: 644da0c1 >2012-12-17 08:33:56,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:56,185 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:33:56,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 92646a2 >2012-12-17 08:33:56,208 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [36fcbeec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:33:56,344 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [5b666f42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:33:56,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4c5e1c10 >2012-12-17 08:33:56,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46843994 >2012-12-17 08:33:56,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:56,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, SpmStopVDSCommand, log id: 46843994 >2012-12-17 08:33:56,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, ResetIrsVDSCommand, log id: 4c5e1c10 >2012-12-17 08:33:56,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 767096ea >2012-12-17 08:33:56,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:56,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 767096ea >2012-12-17 08:33:56,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:33:56,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4580 >2012-12-17 08:33:56,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4580, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44481305 >2012-12-17 08:33:56,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling started: taskId = a1b20d7f-1eca-493c-9595-59836d6d3d90 >2012-12-17 08:33:57,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended: taskId = a1b20d7f-1eca-493c-9595-59836d6d3d90 task status = finished >2012-12-17 08:33:57,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended, spm status: SPM >2012-12-17 08:33:57,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a1b20d7f-1eca-493c-9595-59836d6d3d90), log id: 6a5dfc75 >2012-12-17 08:33:57,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMClearTaskVDSCommand, log id: 6a5dfc75 >2012-12-17 08:33:57,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@40699be1, log id: 44481305 >2012-12-17 08:33:57,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:33:57,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:57,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56669bf8 >2012-12-17 08:33:57,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56669bf8 >2012-12-17 08:33:57,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 92646a2 >2012-12-17 08:33:57,901 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:33:57,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a89e318 >2012-12-17 08:33:57,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:33:57,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ffb8fd >2012-12-17 08:33:57,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ffb8fd >2012-12-17 08:33:57,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a89e318 >2012-12-17 08:33:57,921 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [51f97e37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:34:04,103 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:34:06,722 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [550b34b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:34:06,860 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [44e11955] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:34:06,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 64534e44 >2012-12-17 08:34:06,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d4aa5a1 >2012-12-17 08:34:06,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:07,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, SpmStopVDSCommand, log id: 2d4aa5a1 >2012-12-17 08:34:07,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, ResetIrsVDSCommand, log id: 64534e44 >2012-12-17 08:34:07,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 252600e4 >2012-12-17 08:34:07,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:07,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 252600e4 >2012-12-17 08:34:14,748 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:34:17,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:34:17,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4581 >2012-12-17 08:34:17,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4581, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38700870 >2012-12-17 08:34:17,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling started: taskId = d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f >2012-12-17 08:34:18,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling ended: taskId = d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f task status = finished >2012-12-17 08:34:18,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] spmStart polling ended, spm status: SPM >2012-12-17 08:34:18,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [267551eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d1cf5c6c-4d06-41ec-8e0b-76cd1e30d98f), log id: 5d724dd0 >2012-12-17 08:34:18,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [267551eb] FINISH, HSMClearTaskVDSCommand, log id: 5d724dd0 >2012-12-17 08:34:18,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [267551eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b52c29d, log id: 38700870 >2012-12-17 08:34:18,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:34:18,422 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:34:18,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 23f4b03e >2012-12-17 08:34:18,449 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [267551eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:34:18,668 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [5fbba580] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:34:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67e8911 >2012-12-17 08:34:18,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6765964b >2012-12-17 08:34:18,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:19,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, SpmStopVDSCommand, log id: 6765964b >2012-12-17 08:34:19,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, ResetIrsVDSCommand, log id: 67e8911 >2012-12-17 08:34:19,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 519d812b >2012-12-17 08:34:19,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:19,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 519d812b >2012-12-17 08:34:19,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:34:19,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4582 >2012-12-17 08:34:19,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4582, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d54d0 >2012-12-17 08:34:19,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = 26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4 >2012-12-17 08:34:20,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = 26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4 task status = finished >2012-12-17 08:34:20,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM >2012-12-17 08:34:20,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=26c71c62-1850-4ffe-b7e4-d4e5b7b98eb4), log id: 19c49a18 >2012-12-17 08:34:20,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 19c49a18 >2012-12-17 08:34:20,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3a60aadf, log id: 2d54d0 >2012-12-17 08:34:21,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:34:21,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:21,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c55f486 >2012-12-17 08:34:21,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c55f486 >2012-12-17 08:34:21,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23f4b03e >2012-12-17 08:34:21,711 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:34:21,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72cc3a29 >2012-12-17 08:34:21,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:21,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b2d2c8b >2012-12-17 08:34:21,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b2d2c8b >2012-12-17 08:34:21,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72cc3a29 >2012-12-17 08:34:21,731 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:34:24,977 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:34:29,520 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1a2cf793] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:34:29,760 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [7f190dc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:34:29,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 427fa9cf >2012-12-17 08:34:29,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57c06c82 >2012-12-17 08:34:29,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:30,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStopVDSCommand, log id: 57c06c82 >2012-12-17 08:34:30,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, ResetIrsVDSCommand, log id: 427fa9cf >2012-12-17 08:34:30,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11e573e >2012-12-17 08:34:30,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:30,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11e573e >2012-12-17 08:34:35,210 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:34:41,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:34:41,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4583 >2012-12-17 08:34:41,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4583, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4303267a >2012-12-17 08:34:41,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling started: taskId = 68d8add5-05f3-43e9-850e-74f29c1d1869 >2012-12-17 08:34:42,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling ended: taskId = 68d8add5-05f3-43e9-850e-74f29c1d1869 task status = finished >2012-12-17 08:34:42,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] spmStart polling ended, spm status: SPM >2012-12-17 08:34:42,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2c13a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=68d8add5-05f3-43e9-850e-74f29c1d1869), log id: 59ae6494 >2012-12-17 08:34:42,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, HSMClearTaskVDSCommand, log id: 59ae6494 >2012-12-17 08:34:42,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2c13a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@484d5e8b, log id: 4303267a >2012-12-17 08:34:42,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:34:42,202 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:34:42,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f20e7da >2012-12-17 08:34:42,230 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2c13a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:34:42,370 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [170be1f2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:34:42,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f6d2ed >2012-12-17 08:34:42,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f16dd59 >2012-12-17 08:34:42,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:42,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, SpmStopVDSCommand, log id: 6f16dd59 >2012-12-17 08:34:42,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, ResetIrsVDSCommand, log id: 4f6d2ed >2012-12-17 08:34:42,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4bbc53e3 >2012-12-17 08:34:42,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [170be1f2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:42,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [170be1f2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4bbc53e3 >2012-12-17 08:34:42,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:34:42,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4584 >2012-12-17 08:34:42,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4584, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a64197d >2012-12-17 08:34:42,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling started: taskId = f85619b9-5674-41aa-a188-b618b3588afe >2012-12-17 08:34:43,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling ended: taskId = f85619b9-5674-41aa-a188-b618b3588afe task status = finished >2012-12-17 08:34:43,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] spmStart polling ended, spm status: SPM >2012-12-17 08:34:43,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f85619b9-5674-41aa-a188-b618b3588afe), log id: 3855735 >2012-12-17 08:34:43,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMClearTaskVDSCommand, log id: 3855735 >2012-12-17 08:34:43,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27205a61, log id: 3a64197d >2012-12-17 08:34:43,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:34:43,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:43,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67f5506 >2012-12-17 08:34:43,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67f5506 >2012-12-17 08:34:43,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f20e7da >2012-12-17 08:34:43,894 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:34:43,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11a7003f >2012-12-17 08:34:43,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:43,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a055145 >2012-12-17 08:34:43,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a055145 >2012-12-17 08:34:43,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [1981e345] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11a7003f >2012-12-17 08:34:43,913 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [1981e345] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:34:46,589 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:34:52,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [346cd466] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:34:52,888 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [39b86af9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:34:52,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8a2feb5 >2012-12-17 08:34:52,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 525f3d86 >2012-12-17 08:34:52,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:53,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, SpmStopVDSCommand, log id: 525f3d86 >2012-12-17 08:34:53,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, ResetIrsVDSCommand, log id: 8a2feb5 >2012-12-17 08:34:53,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fa306bf >2012-12-17 08:34:53,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:34:53,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fa306bf >2012-12-17 08:34:56,806 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Checking autorecoverable hosts >2012-12-17 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Autorecovering 0 hosts >2012-12-17 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Checking autorecoverable hosts done >2012-12-17 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [2edbd860] Autorecovering storage domains is disabled, skipping >2012-12-17 08:35:03,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:03,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4585 >2012-12-17 08:35:03,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4585, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 648d3b16 >2012-12-17 08:35:03,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling started: taskId = 40d4a8a7-5275-445f-a812-f59d3916fad3 >2012-12-17 08:35:04,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling ended: taskId = 40d4a8a7-5275-445f-a812-f59d3916fad3 task status = finished >2012-12-17 08:35:04,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] spmStart polling ended, spm status: SPM >2012-12-17 08:35:04,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=40d4a8a7-5275-445f-a812-f59d3916fad3), log id: 74b3b0b >2012-12-17 08:35:04,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, HSMClearTaskVDSCommand, log id: 74b3b0b >2012-12-17 08:35:04,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [73dddeb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c1152ea, log id: 648d3b16 >2012-12-17 08:35:04,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:04,868 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:35:04,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a273f21 >2012-12-17 08:35:04,895 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [73dddeb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:35:05,038 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [39a5c05e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:35:05,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7eae072e >2012-12-17 08:35:05,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78e3a230 >2012-12-17 08:35:05,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:05,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStopVDSCommand, log id: 78e3a230 >2012-12-17 08:35:06,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, ResetIrsVDSCommand, log id: 7eae072e >2012-12-17 08:35:06,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 106387e1 >2012-12-17 08:35:06,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:06,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 106387e1 >2012-12-17 08:35:06,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:06,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4586 >2012-12-17 08:35:06,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4586, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e37d2f6 >2012-12-17 08:35:06,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling started: taskId = 5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327 >2012-12-17 08:35:07,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended: taskId = 5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327 task status = finished >2012-12-17 08:35:07,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended, spm status: SPM >2012-12-17 08:35:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f4f10b3-7a51-4b4c-8f8c-5e7651d4e327), log id: 75fa42b7 >2012-12-17 08:35:07,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMClearTaskVDSCommand, log id: 75fa42b7 >2012-12-17 08:35:07,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13238e4, log id: 7e37d2f6 >2012-12-17 08:35:07,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:07,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:07,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 451a1a97 >2012-12-17 08:35:07,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 451a1a97 >2012-12-17 08:35:07,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a273f21 >2012-12-17 08:35:07,894 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:07,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34c7d2f2 >2012-12-17 08:35:07,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:07,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5af96ac9 >2012-12-17 08:35:07,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5af96ac9 >2012-12-17 08:35:07,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34c7d2f2 >2012-12-17 08:35:07,913 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:16,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57c34106] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:35:16,908 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [1390e512] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:35:16,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17aa58a1 >2012-12-17 08:35:16,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4182df9a >2012-12-17 08:35:16,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:17,198 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:35:17,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, SpmStopVDSCommand, log id: 4182df9a >2012-12-17 08:35:17,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, ResetIrsVDSCommand, log id: 17aa58a1 >2012-12-17 08:35:17,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75dcc2e2 >2012-12-17 08:35:17,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:17,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75dcc2e2 >2012-12-17 08:35:27,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:27,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4587 >2012-12-17 08:35:27,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4587, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 530d461f >2012-12-17 08:35:27,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling started: taskId = 988539f1-3971-4845-a874-036ab6a601aa >2012-12-17 08:35:28,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended: taskId = 988539f1-3971-4845-a874-036ab6a601aa task status = finished >2012-12-17 08:35:28,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] spmStart polling ended, spm status: SPM >2012-12-17 08:35:28,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=988539f1-3971-4845-a874-036ab6a601aa), log id: 3168740a >2012-12-17 08:35:28,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, HSMClearTaskVDSCommand, log id: 3168740a >2012-12-17 08:35:28,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [8c4c643] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@409a55dd, log id: 530d461f >2012-12-17 08:35:28,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:28,556 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:35:28,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f9ca247 >2012-12-17 08:35:28,574 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [8c4c643] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:35:28,710 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [653dcf69] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:35:28,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 549e38a2 >2012-12-17 08:35:28,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 762956ff >2012-12-17 08:35:28,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:28,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, SpmStopVDSCommand, log id: 762956ff >2012-12-17 08:35:28,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, ResetIrsVDSCommand, log id: 549e38a2 >2012-12-17 08:35:28,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f7aba17 >2012-12-17 08:35:28,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:28,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f7aba17 >2012-12-17 08:35:28,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:28,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4588 >2012-12-17 08:35:28,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4588, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c629138 >2012-12-17 08:35:29,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling started: taskId = 65e66282-cb57-486b-93a2-e2587a521e6f >2012-12-17 08:35:30,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended: taskId = 65e66282-cb57-486b-93a2-e2587a521e6f task status = finished >2012-12-17 08:35:30,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] spmStart polling ended, spm status: SPM >2012-12-17 08:35:30,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=65e66282-cb57-486b-93a2-e2587a521e6f), log id: 1c59b6a3 >2012-12-17 08:35:30,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMClearTaskVDSCommand, log id: 1c59b6a3 >2012-12-17 08:35:30,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45e595df, log id: c629138 >2012-12-17 08:35:30,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:30,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:30,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5573ac6e >2012-12-17 08:35:30,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5573ac6e >2012-12-17 08:35:30,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f9ca247 >2012-12-17 08:35:30,132 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:30,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3627b9d9 >2012-12-17 08:35:30,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:30,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 463d929c >2012-12-17 08:35:30,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 463d929c >2012-12-17 08:35:30,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [2f70811] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3627b9d9 >2012-12-17 08:35:30,153 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [2f70811] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:37,602 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:35:39,033 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [323f195a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:35:39,309 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:35:39,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52fecd18 >2012-12-17 08:35:39,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5272d96a >2012-12-17 08:35:39,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:39,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, SpmStopVDSCommand, log id: 5272d96a >2012-12-17 08:35:39,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, ResetIrsVDSCommand, log id: 52fecd18 >2012-12-17 08:35:39,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56e4aea2 >2012-12-17 08:35:39,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:39,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56e4aea2 >2012-12-17 08:35:47,857 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:35:49,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:49,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4589 >2012-12-17 08:35:49,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4589, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2569beda >2012-12-17 08:35:49,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling started: taskId = ebea22bc-19b0-48be-a460-f5d9da4b2d04 >2012-12-17 08:35:50,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling ended: taskId = ebea22bc-19b0-48be-a460-f5d9da4b2d04 task status = finished >2012-12-17 08:35:50,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] spmStart polling ended, spm status: SPM >2012-12-17 08:35:50,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ebea22bc-19b0-48be-a460-f5d9da4b2d04), log id: 164a5143 >2012-12-17 08:35:50,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, HSMClearTaskVDSCommand, log id: 164a5143 >2012-12-17 08:35:50,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [69e82ff1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@691d1123, log id: 2569beda >2012-12-17 08:35:50,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:51,426 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:35:51,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12b28511 >2012-12-17 08:35:51,593 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69e82ff1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:35:51,721 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [369f35cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:35:51,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44925485 >2012-12-17 08:35:51,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 446cf875 >2012-12-17 08:35:51,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:51,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, SpmStopVDSCommand, log id: 446cf875 >2012-12-17 08:35:51,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, ResetIrsVDSCommand, log id: 44925485 >2012-12-17 08:35:51,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9400c78 >2012-12-17 08:35:51,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:51,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9400c78 >2012-12-17 08:35:52,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:35:52,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4590 >2012-12-17 08:35:52,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4590, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62d399eb >2012-12-17 08:35:52,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling started: taskId = 3f13cba0-32a4-4a35-abb1-9f606bb44bc2 >2012-12-17 08:35:53,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended: taskId = 3f13cba0-32a4-4a35-abb1-9f606bb44bc2 task status = finished >2012-12-17 08:35:53,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] spmStart polling ended, spm status: SPM >2012-12-17 08:35:53,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3f13cba0-32a4-4a35-abb1-9f606bb44bc2), log id: 11628e8e >2012-12-17 08:35:53,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMClearTaskVDSCommand, log id: 11628e8e >2012-12-17 08:35:53,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d7d158c, log id: 62d399eb >2012-12-17 08:35:53,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:35:53,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:53,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b301a20 >2012-12-17 08:35:53,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b301a20 >2012-12-17 08:35:53,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12b28511 >2012-12-17 08:35:53,189 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:53,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20f279cd >2012-12-17 08:35:53,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:35:53,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72c887b >2012-12-17 08:35:53,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72c887b >2012-12-17 08:35:53,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [59932b3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20f279cd >2012-12-17 08:35:53,209 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [59932b3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:35:58,036 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:02,080 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [65976fd0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:02,228 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [639cabd2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:02,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 118c427f >2012-12-17 08:36:02,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d05b855 >2012-12-17 08:36:02,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:02,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, SpmStopVDSCommand, log id: 1d05b855 >2012-12-17 08:36:02,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, ResetIrsVDSCommand, log id: 118c427f >2012-12-17 08:36:02,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4a3da75e >2012-12-17 08:36:02,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:02,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4a3da75e >2012-12-17 08:36:08,241 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:12,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:12,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4591 >2012-12-17 08:36:12,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4591, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47fd183b >2012-12-17 08:36:12,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling started: taskId = 57f689e4-44ea-40d3-865a-a577d1377028 >2012-12-17 08:36:13,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling ended: taskId = 57f689e4-44ea-40d3-865a-a577d1377028 task status = finished >2012-12-17 08:36:13,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] spmStart polling ended, spm status: SPM >2012-12-17 08:36:13,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [70373480] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57f689e4-44ea-40d3-865a-a577d1377028), log id: 2472cb60 >2012-12-17 08:36:13,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, HSMClearTaskVDSCommand, log id: 2472cb60 >2012-12-17 08:36:13,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [70373480] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c861e51, log id: 47fd183b >2012-12-17 08:36:13,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:36:13,698 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:36:13,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f12c5d1 >2012-12-17 08:36:13,741 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70373480] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:13,873 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [1daf49b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:13,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13da28bf >2012-12-17 08:36:13,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25f5444a >2012-12-17 08:36:13,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:14,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, SpmStopVDSCommand, log id: 25f5444a >2012-12-17 08:36:14,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, ResetIrsVDSCommand, log id: 13da28bf >2012-12-17 08:36:14,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c78c5ef >2012-12-17 08:36:14,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:14,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c78c5ef >2012-12-17 08:36:14,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:14,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4592 >2012-12-17 08:36:14,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4592, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26effec0 >2012-12-17 08:36:14,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = b24585df-ea97-4806-bbdb-31e52e15122d >2012-12-17 08:36:15,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = b24585df-ea97-4806-bbdb-31e52e15122d task status = finished >2012-12-17 08:36:15,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM >2012-12-17 08:36:15,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b24585df-ea97-4806-bbdb-31e52e15122d), log id: 260545d4 >2012-12-17 08:36:15,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 260545d4 >2012-12-17 08:36:15,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@242b74cc, log id: 26effec0 >2012-12-17 08:36:15,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:36:15,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:15,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 422df258 >2012-12-17 08:36:15,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 422df258 >2012-12-17 08:36:15,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f12c5d1 >2012-12-17 08:36:15,329 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:36:15,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 546d966d >2012-12-17 08:36:15,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:15,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cc0d581 >2012-12-17 08:36:15,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cc0d581 >2012-12-17 08:36:15,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 546d966d >2012-12-17 08:36:15,349 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:36:18,504 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:24,223 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4579c6a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:24,361 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4f93d023] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:24,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57d3e7c5 >2012-12-17 08:36:24,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1ec3e2a8 >2012-12-17 08:36:24,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:24,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, SpmStopVDSCommand, log id: 1ec3e2a8 >2012-12-17 08:36:24,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, ResetIrsVDSCommand, log id: 57d3e7c5 >2012-12-17 08:36:24,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ff10101 >2012-12-17 08:36:24,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:24,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ff10101 >2012-12-17 08:36:28,700 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:34,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:34,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4593 >2012-12-17 08:36:34,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4593, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52d1342c >2012-12-17 08:36:34,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling started: taskId = 1c819230-12f8-4001-83f1-0a3a4a0c53bd >2012-12-17 08:36:35,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling ended: taskId = 1c819230-12f8-4001-83f1-0a3a4a0c53bd task status = finished >2012-12-17 08:36:35,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] spmStart polling ended, spm status: SPM >2012-12-17 08:36:35,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1c819230-12f8-4001-83f1-0a3a4a0c53bd), log id: 21aba7e5 >2012-12-17 08:36:35,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, HSMClearTaskVDSCommand, log id: 21aba7e5 >2012-12-17 08:36:35,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [69d0a8ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c9bbca8, log id: 52d1342c >2012-12-17 08:36:36,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:36:36,261 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:36:36,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b380e7b >2012-12-17 08:36:36,283 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [69d0a8ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:36,449 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [24452d4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:36,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 427cea7e >2012-12-17 08:36:36,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26c9613b >2012-12-17 08:36:36,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:36,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, SpmStopVDSCommand, log id: 26c9613b >2012-12-17 08:36:36,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, ResetIrsVDSCommand, log id: 427cea7e >2012-12-17 08:36:36,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 239678d0 >2012-12-17 08:36:36,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:36,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 239678d0 >2012-12-17 08:36:36,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:36,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4594 >2012-12-17 08:36:36,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4594, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23f375e8 >2012-12-17 08:36:36,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling started: taskId = 1a78f8e1-e9c1-4195-8414-aeebec9a07ed >2012-12-17 08:36:37,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling ended: taskId = 1a78f8e1-e9c1-4195-8414-aeebec9a07ed task status = finished >2012-12-17 08:36:37,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] spmStart polling ended, spm status: SPM >2012-12-17 08:36:37,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1a78f8e1-e9c1-4195-8414-aeebec9a07ed), log id: 73bf7cbf >2012-12-17 08:36:37,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMClearTaskVDSCommand, log id: 73bf7cbf >2012-12-17 08:36:37,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20971af7, log id: 23f375e8 >2012-12-17 08:36:37,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:36:37,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:37,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ec55324 >2012-12-17 08:36:37,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ec55324 >2012-12-17 08:36:37,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b380e7b >2012-12-17 08:36:37,919 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:36:37,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14f32617 >2012-12-17 08:36:37,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:37,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5985a781 >2012-12-17 08:36:37,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5985a781 >2012-12-17 08:36:37,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2edbd860] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14f32617 >2012-12-17 08:36:37,940 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2edbd860] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:36:38,948 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:46,796 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [676a97fd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:46,949 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [4232612d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:46,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3904c3d3 >2012-12-17 08:36:46,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51e8c35c >2012-12-17 08:36:47,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:47,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, SpmStopVDSCommand, log id: 51e8c35c >2012-12-17 08:36:47,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, ResetIrsVDSCommand, log id: 3904c3d3 >2012-12-17 08:36:47,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4232612d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5eacdb29 >2012-12-17 08:36:47,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:47,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4232612d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5eacdb29 >2012-12-17 08:36:49,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:36:57,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:57,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4595 >2012-12-17 08:36:57,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4595, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a8f17c1 >2012-12-17 08:36:57,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling started: taskId = 8fd3289e-496c-4669-b7b8-6827775c6d66 >2012-12-17 08:36:58,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended: taskId = 8fd3289e-496c-4669-b7b8-6827775c6d66 task status = finished >2012-12-17 08:36:58,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] spmStart polling ended, spm status: SPM >2012-12-17 08:36:58,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8fd3289e-496c-4669-b7b8-6827775c6d66), log id: 71f2d1a7 >2012-12-17 08:36:58,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, HSMClearTaskVDSCommand, log id: 71f2d1a7 >2012-12-17 08:36:58,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [25ec3656] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ca26562, log id: a8f17c1 >2012-12-17 08:36:58,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:36:58,632 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:36:58,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6832f6d >2012-12-17 08:36:58,652 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [25ec3656] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:36:58,775 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [1ee6f70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:36:58,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 452b64e2 >2012-12-17 08:36:58,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 224076e4 >2012-12-17 08:36:58,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:58,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, SpmStopVDSCommand, log id: 224076e4 >2012-12-17 08:36:59,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, ResetIrsVDSCommand, log id: 452b64e2 >2012-12-17 08:36:59,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e9f1412 >2012-12-17 08:36:59,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:36:59,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [1ee6f70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e9f1412 >2012-12-17 08:36:59,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:36:59,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4596 >2012-12-17 08:36:59,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4596, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 191bfd63 >2012-12-17 08:36:59,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling started: taskId = bc21521f-f82b-4e40-9a8d-c569f9e8e2e5 >2012-12-17 08:37:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended: taskId = bc21521f-f82b-4e40-9a8d-c569f9e8e2e5 task status = finished >2012-12-17 08:37:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended, spm status: SPM >2012-12-17 08:37:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc21521f-f82b-4e40-9a8d-c569f9e8e2e5), log id: 172789fc >2012-12-17 08:37:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMClearTaskVDSCommand, log id: 172789fc >2012-12-17 08:37:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24d40b44, log id: 191bfd63 >2012-12-17 08:37:00,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:37:00,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d87ddf0 >2012-12-17 08:37:00,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d87ddf0 >2012-12-17 08:37:00,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6832f6d >2012-12-17 08:37:00,239 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:00,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22fbd71f >2012-12-17 08:37:00,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40536ce0 >2012-12-17 08:37:00,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40536ce0 >2012-12-17 08:37:00,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22fbd71f >2012-12-17 08:37:00,259 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [169ff372] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:09,137 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [27ff9aa4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:37:09,274 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:37:09,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67920a70 >2012-12-17 08:37:09,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5c4905a7 >2012-12-17 08:37:09,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:09,553 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:37:09,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SpmStopVDSCommand, log id: 5c4905a7 >2012-12-17 08:37:09,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, ResetIrsVDSCommand, log id: 67920a70 >2012-12-17 08:37:09,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73f4aaea >2012-12-17 08:37:09,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:09,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73f4aaea >2012-12-17 08:37:19,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:37:19,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4597 >2012-12-17 08:37:19,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4597, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f67cac3 >2012-12-17 08:37:19,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling started: taskId = 27dba14f-d616-43e6-bb20-4eb68e6de390 >2012-12-17 08:37:20,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling ended: taskId = 27dba14f-d616-43e6-bb20-4eb68e6de390 task status = finished >2012-12-17 08:37:20,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] spmStart polling ended, spm status: SPM >2012-12-17 08:37:20,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [733ac637] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=27dba14f-d616-43e6-bb20-4eb68e6de390), log id: 2e1493bf >2012-12-17 08:37:20,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [733ac637] FINISH, HSMClearTaskVDSCommand, log id: 2e1493bf >2012-12-17 08:37:20,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [733ac637] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48eff83b, log id: 4f67cac3 >2012-12-17 08:37:20,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:37:21,142 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:37:21,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5194d37f >2012-12-17 08:37:21,181 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [733ac637] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:37:21,361 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [44f1dfe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:37:21,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6910814d >2012-12-17 08:37:21,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2dcfd2cb >2012-12-17 08:37:21,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:21,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, SpmStopVDSCommand, log id: 2dcfd2cb >2012-12-17 08:37:21,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, ResetIrsVDSCommand, log id: 6910814d >2012-12-17 08:37:21,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12e6c080 >2012-12-17 08:37:21,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:21,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12e6c080 >2012-12-17 08:37:21,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:37:21,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4598 >2012-12-17 08:37:21,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4598, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e284867 >2012-12-17 08:37:21,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = ad1d8e50-95da-47f0-a115-f593652f37ec >2012-12-17 08:37:22,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = ad1d8e50-95da-47f0-a115-f593652f37ec task status = finished >2012-12-17 08:37:22,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM >2012-12-17 08:37:22,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ad1d8e50-95da-47f0-a115-f593652f37ec), log id: 58033b29 >2012-12-17 08:37:22,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 58033b29 >2012-12-17 08:37:22,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c212de3, log id: 7e284867 >2012-12-17 08:37:22,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:37:22,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:22,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d7605a5 >2012-12-17 08:37:22,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d7605a5 >2012-12-17 08:37:22,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5194d37f >2012-12-17 08:37:22,816 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:22,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f6cd3fc >2012-12-17 08:37:22,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:22,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52d6441a >2012-12-17 08:37:22,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52d6441a >2012-12-17 08:37:22,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f6cd3fc >2012-12-17 08:37:22,835 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:29,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:37:31,721 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [47f0782e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:37:31,803 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [50db5e16] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:37:31,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79787e70 >2012-12-17 08:37:31,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5307046d >2012-12-17 08:37:31,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:32,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, SpmStopVDSCommand, log id: 5307046d >2012-12-17 08:37:32,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, ResetIrsVDSCommand, log id: 79787e70 >2012-12-17 08:37:32,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cb1ca87 >2012-12-17 08:37:32,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:32,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cb1ca87 >2012-12-17 08:37:40,145 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:37:42,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:37:42,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4599 >2012-12-17 08:37:42,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4599, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c1d3d63 >2012-12-17 08:37:42,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling started: taskId = d7641631-09d0-476d-bcf0-4712138f71fb >2012-12-17 08:37:43,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling ended: taskId = d7641631-09d0-476d-bcf0-4712138f71fb task status = finished >2012-12-17 08:37:43,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] spmStart polling ended, spm status: SPM >2012-12-17 08:37:43,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7641631-09d0-476d-bcf0-4712138f71fb), log id: 41187a7b >2012-12-17 08:37:43,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] FINISH, HSMClearTaskVDSCommand, log id: 41187a7b >2012-12-17 08:37:43,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [318b1f4f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e1142a0, log id: 4c1d3d63 >2012-12-17 08:37:43,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:37:43,915 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:37:43,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1456527c >2012-12-17 08:37:43,943 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [318b1f4f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:37:44,076 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [4beddd99] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:37:44,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71a07f19 >2012-12-17 08:37:44,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4115c712 >2012-12-17 08:37:44,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:44,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, SpmStopVDSCommand, log id: 4115c712 >2012-12-17 08:37:44,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, ResetIrsVDSCommand, log id: 71a07f19 >2012-12-17 08:37:44,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a011164 >2012-12-17 08:37:44,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:44,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4beddd99] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a011164 >2012-12-17 08:37:44,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:37:44,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4600 >2012-12-17 08:37:44,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4600, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 43d0f323 >2012-12-17 08:37:44,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling started: taskId = 7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0 >2012-12-17 08:37:45,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended: taskId = 7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0 task status = finished >2012-12-17 08:37:45,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended, spm status: SPM >2012-12-17 08:37:45,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7374690a-afeb-4cf1-93cc-3f5fe1d6c5f0), log id: 516ec621 >2012-12-17 08:37:45,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMClearTaskVDSCommand, log id: 516ec621 >2012-12-17 08:37:45,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f91ca4b, log id: 43d0f323 >2012-12-17 08:37:45,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:37:46,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:46,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 220c6a2 >2012-12-17 08:37:46,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 220c6a2 >2012-12-17 08:37:46,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1456527c >2012-12-17 08:37:46,072 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:46,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12a6da07 >2012-12-17 08:37:46,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:46,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c43cadd >2012-12-17 08:37:46,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c43cadd >2012-12-17 08:37:46,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12a6da07 >2012-12-17 08:37:46,093 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [cdacbd1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:37:50,966 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:37:54,481 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1981e345] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:37:54,650 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [5266d504] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:37:54,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 15389550 >2012-12-17 08:37:54,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e04a16 >2012-12-17 08:37:54,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:55,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, SpmStopVDSCommand, log id: 7e04a16 >2012-12-17 08:37:55,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, ResetIrsVDSCommand, log id: 15389550 >2012-12-17 08:37:55,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5266d504] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40787858 >2012-12-17 08:37:55,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:37:55,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5266d504] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40787858 >2012-12-17 08:38:01,259 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:38:06,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:06,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4601 >2012-12-17 08:38:06,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4601, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e50f11c >2012-12-17 08:38:06,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling started: taskId = 14227764-0b58-4f8b-ad3f-09edb98a1c8d >2012-12-17 08:38:07,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended: taskId = 14227764-0b58-4f8b-ad3f-09edb98a1c8d task status = finished >2012-12-17 08:38:07,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] spmStart polling ended, spm status: SPM >2012-12-17 08:38:07,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14227764-0b58-4f8b-ad3f-09edb98a1c8d), log id: 6272c868 >2012-12-17 08:38:07,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, HSMClearTaskVDSCommand, log id: 6272c868 >2012-12-17 08:38:07,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [54cb0ab7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dc35656, log id: 1e50f11c >2012-12-17 08:38:07,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:07,640 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:38:07,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2aaf26a9 >2012-12-17 08:38:07,674 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [54cb0ab7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:38:07,880 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [56986c52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:38:07,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1cf53cdf >2012-12-17 08:38:07,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4cfea686 >2012-12-17 08:38:08,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:08,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, SpmStopVDSCommand, log id: 4cfea686 >2012-12-17 08:38:08,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, ResetIrsVDSCommand, log id: 1cf53cdf >2012-12-17 08:38:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 734c2cf5 >2012-12-17 08:38:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 734c2cf5 >2012-12-17 08:38:08,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:08,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4602 >2012-12-17 08:38:08,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4602, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d52348e >2012-12-17 08:38:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = ea11719f-75d5-4c8a-88b7-05492448ddbf >2012-12-17 08:38:09,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = ea11719f-75d5-4c8a-88b7-05492448ddbf task status = finished >2012-12-17 08:38:09,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM >2012-12-17 08:38:09,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ea11719f-75d5-4c8a-88b7-05492448ddbf), log id: dcf542d >2012-12-17 08:38:09,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: dcf542d >2012-12-17 08:38:09,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a6b50e2, log id: 1d52348e >2012-12-17 08:38:09,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:09,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:09,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52c8436e >2012-12-17 08:38:09,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52c8436e >2012-12-17 08:38:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2aaf26a9 >2012-12-17 08:38:09,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:38:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 16a5fbeb >2012-12-17 08:38:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:09,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2aaa7331 >2012-12-17 08:38:09,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2aaa7331 >2012-12-17 08:38:09,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 16a5fbeb >2012-12-17 08:38:09,469 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:38:11,553 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:38:18,389 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2afeec77] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:38:18,542 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [1f4324c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:38:18,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b062fbd >2012-12-17 08:38:18,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e542f3b >2012-12-17 08:38:18,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:18,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStopVDSCommand, log id: 2e542f3b >2012-12-17 08:38:18,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, ResetIrsVDSCommand, log id: b062fbd >2012-12-17 08:38:18,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28a27b45 >2012-12-17 08:38:18,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:18,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28a27b45 >2012-12-17 08:38:21,754 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:38:28,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:28,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4603 >2012-12-17 08:38:28,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4603, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6326ca34 >2012-12-17 08:38:28,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling started: taskId = a80b6299-28e2-43e9-a6e3-0d18c0814ad8 >2012-12-17 08:38:29,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling ended: taskId = a80b6299-28e2-43e9-a6e3-0d18c0814ad8 task status = finished >2012-12-17 08:38:29,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] spmStart polling ended, spm status: SPM >2012-12-17 08:38:29,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a80b6299-28e2-43e9-a6e3-0d18c0814ad8), log id: 64a4d108 >2012-12-17 08:38:29,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] FINISH, HSMClearTaskVDSCommand, log id: 64a4d108 >2012-12-17 08:38:29,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [6e64c0c8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9797d7, log id: 6326ca34 >2012-12-17 08:38:30,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:30,048 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:38:30,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 578fe9e8 >2012-12-17 08:38:30,077 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [6e64c0c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:38:30,205 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [b3b23b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:38:30,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79ae25f7 >2012-12-17 08:38:30,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ed09b6a >2012-12-17 08:38:30,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:30,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStopVDSCommand, log id: 5ed09b6a >2012-12-17 08:38:30,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, ResetIrsVDSCommand, log id: 79ae25f7 >2012-12-17 08:38:30,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19cf0981 >2012-12-17 08:38:30,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:30,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19cf0981 >2012-12-17 08:38:31,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:31,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4604 >2012-12-17 08:38:31,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4604, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d81057a >2012-12-17 08:38:31,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling started: taskId = 0bc9cc0d-4c7b-4371-89da-b28ea9b5a412 >2012-12-17 08:38:32,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling ended: taskId = 0bc9cc0d-4c7b-4371-89da-b28ea9b5a412 task status = finished >2012-12-17 08:38:32,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] spmStart polling ended, spm status: SPM >2012-12-17 08:38:32,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0bc9cc0d-4c7b-4371-89da-b28ea9b5a412), log id: 89689e0 >2012-12-17 08:38:32,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMClearTaskVDSCommand, log id: 89689e0 >2012-12-17 08:38:32,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5335d587, log id: 5d81057a >2012-12-17 08:38:32,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:32,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:32,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 190546fe >2012-12-17 08:38:32,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 190546fe >2012-12-17 08:38:32,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 578fe9e8 >2012-12-17 08:38:32,511 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:38:32,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2942d4b0 >2012-12-17 08:38:32,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:32,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a7cddae >2012-12-17 08:38:32,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a7cddae >2012-12-17 08:38:32,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [95f5063] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2942d4b0 >2012-12-17 08:38:32,542 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [95f5063] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:38:41,584 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2f70811] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:38:41,750 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [41698d74] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:38:41,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 284bfdb3 >2012-12-17 08:38:41,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 406d69c7 >2012-12-17 08:38:41,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:42,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SpmStopVDSCommand, log id: 406d69c7 >2012-12-17 08:38:42,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, ResetIrsVDSCommand, log id: 284bfdb3 >2012-12-17 08:38:42,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a9bd265 >2012-12-17 08:38:42,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:42,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a9bd265 >2012-12-17 08:38:42,164 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:38:52,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:52,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4605 >2012-12-17 08:38:52,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4605, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23022fd2 >2012-12-17 08:38:52,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling started: taskId = 449b3ba0-4929-4634-9128-0a425c3f5874 >2012-12-17 08:38:53,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling ended: taskId = 449b3ba0-4929-4634-9128-0a425c3f5874 task status = finished >2012-12-17 08:38:53,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] spmStart polling ended, spm status: SPM >2012-12-17 08:38:53,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=449b3ba0-4929-4634-9128-0a425c3f5874), log id: 6ef11e6d >2012-12-17 08:38:53,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, HSMClearTaskVDSCommand, log id: 6ef11e6d >2012-12-17 08:38:53,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [39b86af9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1804bf02, log id: 23022fd2 >2012-12-17 08:38:53,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:53,425 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:38:53,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4136adc8 >2012-12-17 08:38:53,461 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [39b86af9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:38:53,545 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [25db9572] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:38:53,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 724b53 >2012-12-17 08:38:53,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57a8d71 >2012-12-17 08:38:53,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:53,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, SpmStopVDSCommand, log id: 57a8d71 >2012-12-17 08:38:53,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, ResetIrsVDSCommand, log id: 724b53 >2012-12-17 08:38:53,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [25db9572] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49c8b150 >2012-12-17 08:38:53,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:53,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [25db9572] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49c8b150 >2012-12-17 08:38:53,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:38:53,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4606 >2012-12-17 08:38:53,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4606, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7164aa23 >2012-12-17 08:38:53,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling started: taskId = e9704d9e-eeed-4a50-a97c-07c08f8f2325 >2012-12-17 08:38:54,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended: taskId = e9704d9e-eeed-4a50-a97c-07c08f8f2325 task status = finished >2012-12-17 08:38:54,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] spmStart polling ended, spm status: SPM >2012-12-17 08:38:54,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9704d9e-eeed-4a50-a97c-07c08f8f2325), log id: 55598f7a >2012-12-17 08:38:54,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMClearTaskVDSCommand, log id: 55598f7a >2012-12-17 08:38:54,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dd65fa2, log id: 7164aa23 >2012-12-17 08:38:54,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:38:54,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:54,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b7cfaef >2012-12-17 08:38:55,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b7cfaef >2012-12-17 08:38:55,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4136adc8 >2012-12-17 08:38:55,007 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:38:55,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f1080fc >2012-12-17 08:38:55,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:38:55,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 767259a7 >2012-12-17 08:38:55,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 767259a7 >2012-12-17 08:38:55,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [406132e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f1080fc >2012-12-17 08:38:55,027 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [406132e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:39:02,629 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:03,872 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [59932b3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:39:04,027 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [1a6bc63] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:39:04,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a2471d7 >2012-12-17 08:39:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34aff74e >2012-12-17 08:39:04,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:04,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, SpmStopVDSCommand, log id: 34aff74e >2012-12-17 08:39:04,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, ResetIrsVDSCommand, log id: 4a2471d7 >2012-12-17 08:39:04,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 750565d7 >2012-12-17 08:39:04,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:04,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [1a6bc63] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 750565d7 >2012-12-17 08:39:12,868 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:14,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:39:14,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4607 >2012-12-17 08:39:14,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4607, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20dcbe8d >2012-12-17 08:39:14,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling started: taskId = 51c7c47e-e18e-4f82-ab0d-06f9a9d592d4 >2012-12-17 08:39:15,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling ended: taskId = 51c7c47e-e18e-4f82-ab0d-06f9a9d592d4 task status = finished >2012-12-17 08:39:15,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] spmStart polling ended, spm status: SPM >2012-12-17 08:39:15,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=51c7c47e-e18e-4f82-ab0d-06f9a9d592d4), log id: 42c79c9e >2012-12-17 08:39:15,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, HSMClearTaskVDSCommand, log id: 42c79c9e >2012-12-17 08:39:15,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [639cabd2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3efb6b52, log id: 20dcbe8d >2012-12-17 08:39:16,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:39:16,114 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:39:16,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20826965 >2012-12-17 08:39:16,140 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [639cabd2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:39:16,246 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [7d3891a6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:39:16,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d7fc4d9 >2012-12-17 08:39:16,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c7ef659 >2012-12-17 08:39:16,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:16,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SpmStopVDSCommand, log id: 3c7ef659 >2012-12-17 08:39:16,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, ResetIrsVDSCommand, log id: 2d7fc4d9 >2012-12-17 08:39:16,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3200bb49 >2012-12-17 08:39:16,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:16,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3200bb49 >2012-12-17 08:39:16,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:39:16,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4608 >2012-12-17 08:39:16,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4608, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bc778bc >2012-12-17 08:39:16,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling started: taskId = a10956fa-88f8-4ede-9e78-33268e3283de >2012-12-17 08:39:17,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling ended: taskId = a10956fa-88f8-4ede-9e78-33268e3283de task status = finished >2012-12-17 08:39:17,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] spmStart polling ended, spm status: SPM >2012-12-17 08:39:17,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a10956fa-88f8-4ede-9e78-33268e3283de), log id: 7936bf4c >2012-12-17 08:39:18,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMClearTaskVDSCommand, log id: 7936bf4c >2012-12-17 08:39:18,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@794954bf, log id: 2bc778bc >2012-12-17 08:39:18,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:39:18,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:18,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1545f3f0 >2012-12-17 08:39:18,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1545f3f0 >2012-12-17 08:39:18,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20826965 >2012-12-17 08:39:18,076 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:39:18,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 23921c79 >2012-12-17 08:39:18,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:18,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25052668 >2012-12-17 08:39:18,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25052668 >2012-12-17 08:39:18,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [a2fcda8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23921c79 >2012-12-17 08:39:18,097 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [a2fcda8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:39:23,134 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:26,958 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [170be1f2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:39:27,065 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [2324f948] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:39:27,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2752da87 >2012-12-17 08:39:27,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3297ea5e >2012-12-17 08:39:27,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:27,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStopVDSCommand, log id: 3297ea5e >2012-12-17 08:39:27,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, ResetIrsVDSCommand, log id: 2752da87 >2012-12-17 08:39:27,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a595460 >2012-12-17 08:39:27,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:27,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a595460 >2012-12-17 08:39:33,340 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:37,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:39:37,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4609 >2012-12-17 08:39:37,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4609, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34d2f5a1 >2012-12-17 08:39:37,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling started: taskId = f436c442-caa9-4d3e-a821-797b0b7349a5 >2012-12-17 08:39:38,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling ended: taskId = f436c442-caa9-4d3e-a821-797b0b7349a5 task status = finished >2012-12-17 08:39:38,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] spmStart polling ended, spm status: SPM >2012-12-17 08:39:38,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f436c442-caa9-4d3e-a821-797b0b7349a5), log id: 2d83166f >2012-12-17 08:39:38,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] FINISH, HSMClearTaskVDSCommand, log id: 2d83166f >2012-12-17 08:39:38,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [1c47e0ef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7935e8c0, log id: 34d2f5a1 >2012-12-17 08:39:38,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:39:38,593 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:39:38,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30515c47 >2012-12-17 08:39:38,658 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1c47e0ef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:39:38,763 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [4595e4ad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:39:38,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dad526a >2012-12-17 08:39:38,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51114c57 >2012-12-17 08:39:38,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:38,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, SpmStopVDSCommand, log id: 51114c57 >2012-12-17 08:39:38,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, ResetIrsVDSCommand, log id: dad526a >2012-12-17 08:39:38,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b1f2562 >2012-12-17 08:39:38,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:38,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b1f2562 >2012-12-17 08:39:39,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:39:39,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4610 >2012-12-17 08:39:39,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4610, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5340a3f >2012-12-17 08:39:39,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling started: taskId = 667c4ddd-9152-407e-8d5c-7c15b1c23ef2 >2012-12-17 08:39:40,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended: taskId = 667c4ddd-9152-407e-8d5c-7c15b1c23ef2 task status = finished >2012-12-17 08:39:40,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] spmStart polling ended, spm status: SPM >2012-12-17 08:39:40,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=667c4ddd-9152-407e-8d5c-7c15b1c23ef2), log id: 5b2a2182 >2012-12-17 08:39:40,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMClearTaskVDSCommand, log id: 5b2a2182 >2012-12-17 08:39:40,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2afe0c90, log id: 5340a3f >2012-12-17 08:39:40,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:39:40,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:40,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14cbfe7d >2012-12-17 08:39:40,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14cbfe7d >2012-12-17 08:39:40,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30515c47 >2012-12-17 08:39:40,227 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:39:40,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5cb0cb35 >2012-12-17 08:39:40,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:40,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ab38693 >2012-12-17 08:39:40,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ab38693 >2012-12-17 08:39:40,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [67020b95] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5cb0cb35 >2012-12-17 08:39:40,247 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [67020b95] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:39:43,558 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:49,082 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2edbd860] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:39:49,172 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [2649fcd5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:39:49,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37efd517 >2012-12-17 08:39:49,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d9d856e >2012-12-17 08:39:49,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:49,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SpmStopVDSCommand, log id: 7d9d856e >2012-12-17 08:39:49,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, ResetIrsVDSCommand, log id: 37efd517 >2012-12-17 08:39:49,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 649069a7 >2012-12-17 08:39:49,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:39:49,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 649069a7 >2012-12-17 08:39:53,817 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:39:59,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:39:59,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4611 >2012-12-17 08:39:59,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4611, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37150937 >2012-12-17 08:39:59,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling started: taskId = 76ae3316-adf5-4a61-9e31-e812274f0b81 >2012-12-17 08:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Checking autorecoverable hosts >2012-12-17 08:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Autorecovering 0 hosts >2012-12-17 08:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Checking autorecoverable hosts done >2012-12-17 08:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [436410e6] Autorecovering storage domains is disabled, skipping >2012-12-17 08:40:00,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling ended: taskId = 76ae3316-adf5-4a61-9e31-e812274f0b81 task status = finished >2012-12-17 08:40:00,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] spmStart polling ended, spm status: SPM >2012-12-17 08:40:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=76ae3316-adf5-4a61-9e31-e812274f0b81), log id: 2275fd22 >2012-12-17 08:40:00,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, HSMClearTaskVDSCommand, log id: 2275fd22 >2012-12-17 08:40:00,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [48aa9372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@703dd4da, log id: 37150937 >2012-12-17 08:40:01,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:01,493 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:40:01,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 796a43dc >2012-12-17 08:40:01,580 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [48aa9372] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:01,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [6e84ec96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:01,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71dad761 >2012-12-17 08:40:01,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: f94d5c2 >2012-12-17 08:40:01,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:01,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SpmStopVDSCommand, log id: f94d5c2 >2012-12-17 08:40:01,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, ResetIrsVDSCommand, log id: 71dad761 >2012-12-17 08:40:02,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1066eaa >2012-12-17 08:40:02,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:02,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1066eaa >2012-12-17 08:40:02,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:40:02,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4612 >2012-12-17 08:40:02,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4612, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56f2fb20 >2012-12-17 08:40:02,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling started: taskId = d371619b-1bc2-48b8-8617-e2b3cfa220f1 >2012-12-17 08:40:03,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling ended: taskId = d371619b-1bc2-48b8-8617-e2b3cfa220f1 task status = finished >2012-12-17 08:40:03,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] spmStart polling ended, spm status: SPM >2012-12-17 08:40:03,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d371619b-1bc2-48b8-8617-e2b3cfa220f1), log id: 34dd01c5 >2012-12-17 08:40:03,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMClearTaskVDSCommand, log id: 34dd01c5 >2012-12-17 08:40:03,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199f47a0, log id: 56f2fb20 >2012-12-17 08:40:03,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:03,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:03,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4c3f6387 >2012-12-17 08:40:03,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4c3f6387 >2012-12-17 08:40:03,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 796a43dc >2012-12-17 08:40:03,263 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:03,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 559f5b2c >2012-12-17 08:40:03,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:03,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1dea2423 >2012-12-17 08:40:03,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1dea2423 >2012-12-17 08:40:03,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [7d3891a6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 559f5b2c >2012-12-17 08:40:03,284 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [7d3891a6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:04,004 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:40:12,093 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [50c114a6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:12,224 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [5231d6f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:12,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1883ac35 >2012-12-17 08:40:12,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 221fc926 >2012-12-17 08:40:12,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:12,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SpmStopVDSCommand, log id: 221fc926 >2012-12-17 08:40:12,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, ResetIrsVDSCommand, log id: 1883ac35 >2012-12-17 08:40:12,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12d0f2ba >2012-12-17 08:40:12,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:12,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12d0f2ba >2012-12-17 08:40:14,289 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:40:22,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:40:22,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4613 >2012-12-17 08:40:22,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4613, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 915e3c7 >2012-12-17 08:40:22,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling started: taskId = 159bc454-800f-4936-9e82-6bbd48ae5e7f >2012-12-17 08:40:23,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling ended: taskId = 159bc454-800f-4936-9e82-6bbd48ae5e7f task status = finished >2012-12-17 08:40:23,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] spmStart polling ended, spm status: SPM >2012-12-17 08:40:23,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=159bc454-800f-4936-9e82-6bbd48ae5e7f), log id: 4543144a >2012-12-17 08:40:23,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, HSMClearTaskVDSCommand, log id: 4543144a >2012-12-17 08:40:23,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4f93d023] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36e929cc, log id: 915e3c7 >2012-12-17 08:40:23,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:23,709 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:40:23,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46900d96 >2012-12-17 08:40:23,727 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4f93d023] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:23,863 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [37434c11] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:23,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7feaddaa >2012-12-17 08:40:23,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 563e2af3 >2012-12-17 08:40:23,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:24,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, SpmStopVDSCommand, log id: 563e2af3 >2012-12-17 08:40:24,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, ResetIrsVDSCommand, log id: 7feaddaa >2012-12-17 08:40:24,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [37434c11] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4846c745 >2012-12-17 08:40:24,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [37434c11] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:24,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [37434c11] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4846c745 >2012-12-17 08:40:24,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:40:24,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4614 >2012-12-17 08:40:24,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4614, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31bc37b2 >2012-12-17 08:40:24,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling started: taskId = b94467bc-d7a9-41a3-973b-fdd25d913fbe >2012-12-17 08:40:25,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended: taskId = b94467bc-d7a9-41a3-973b-fdd25d913fbe task status = finished >2012-12-17 08:40:25,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended, spm status: SPM >2012-12-17 08:40:25,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b94467bc-d7a9-41a3-973b-fdd25d913fbe), log id: 5906b6c7 >2012-12-17 08:40:25,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMClearTaskVDSCommand, log id: 5906b6c7 >2012-12-17 08:40:25,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47c9f07a, log id: 31bc37b2 >2012-12-17 08:40:25,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:25,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:25,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 46d6d585 >2012-12-17 08:40:26,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46d6d585 >2012-12-17 08:40:26,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46900d96 >2012-12-17 08:40:26,004 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:26,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 425c3318 >2012-12-17 08:40:26,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:26,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ca213ac >2012-12-17 08:40:26,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ca213ac >2012-12-17 08:40:26,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 425c3318 >2012-12-17 08:40:26,030 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:34,513 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [5a98cdcf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:34,632 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [71e1da67] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:34,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2decfdeb >2012-12-17 08:40:34,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a492037 >2012-12-17 08:40:34,755 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:40:34,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:35,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, SpmStopVDSCommand, log id: 5a492037 >2012-12-17 08:40:35,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, ResetIrsVDSCommand, log id: 2decfdeb >2012-12-17 08:40:35,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d2bb9a5 >2012-12-17 08:40:35,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:35,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d2bb9a5 >2012-12-17 08:40:46,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:40:46,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4615 >2012-12-17 08:40:46,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4615, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 243f8158 >2012-12-17 08:40:46,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling started: taskId = 7eeba601-5d2c-42cf-ae10-56ca5a93975d >2012-12-17 08:40:47,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling ended: taskId = 7eeba601-5d2c-42cf-ae10-56ca5a93975d task status = finished >2012-12-17 08:40:47,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] spmStart polling ended, spm status: SPM >2012-12-17 08:40:47,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7eeba601-5d2c-42cf-ae10-56ca5a93975d), log id: 3a031217 >2012-12-17 08:40:47,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, HSMClearTaskVDSCommand, log id: 3a031217 >2012-12-17 08:40:47,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [57e0db40] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f98c1c2, log id: 243f8158 >2012-12-17 08:40:47,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:47,562 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:40:47,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6bcbc7af >2012-12-17 08:40:47,573 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [57e0db40] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:47,724 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:47,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 131827 >2012-12-17 08:40:47,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 266a42c6 >2012-12-17 08:40:47,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:48,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, SpmStopVDSCommand, log id: 266a42c6 >2012-12-17 08:40:48,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, ResetIrsVDSCommand, log id: 131827 >2012-12-17 08:40:48,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43580ec8 >2012-12-17 08:40:48,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:48,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43580ec8 >2012-12-17 08:40:48,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:40:48,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4616 >2012-12-17 08:40:48,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4616, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 300e7058 >2012-12-17 08:40:48,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 99d4ef12-f3cb-4b9c-8a24-5645096ca296 >2012-12-17 08:40:49,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 99d4ef12-f3cb-4b9c-8a24-5645096ca296 task status = finished >2012-12-17 08:40:49,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM >2012-12-17 08:40:49,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=99d4ef12-f3cb-4b9c-8a24-5645096ca296), log id: 4cb5bbcf >2012-12-17 08:40:49,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 4cb5bbcf >2012-12-17 08:40:49,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f3d3a61, log id: 300e7058 >2012-12-17 08:40:49,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:40:49,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:49,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a59be09 >2012-12-17 08:40:49,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a59be09 >2012-12-17 08:40:49,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6bcbc7af >2012-12-17 08:40:49,350 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:49,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f27ee9a >2012-12-17 08:40:49,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:49,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63d517b7 >2012-12-17 08:40:49,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63d517b7 >2012-12-17 08:40:49,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f27ee9a >2012-12-17 08:40:49,371 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:40:55,115 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:40:58,181 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e6ad503] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:40:58,286 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [7f520532] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:40:58,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7304e581 >2012-12-17 08:40:58,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d7861cf >2012-12-17 08:40:58,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:58,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStopVDSCommand, log id: 1d7861cf >2012-12-17 08:40:58,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, ResetIrsVDSCommand, log id: 7304e581 >2012-12-17 08:40:58,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 94053ea >2012-12-17 08:40:58,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:40:58,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 94053ea >2012-12-17 08:41:06,529 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:41:08,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:08,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4617 >2012-12-17 08:41:08,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4617, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48b82caa >2012-12-17 08:41:08,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling started: taskId = 8b6bd3d9-d57e-415d-a220-437449316252 >2012-12-17 08:41:09,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling ended: taskId = 8b6bd3d9-d57e-415d-a220-437449316252 task status = finished >2012-12-17 08:41:09,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] spmStart polling ended, spm status: SPM >2012-12-17 08:41:09,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b6bd3d9-d57e-415d-a220-437449316252), log id: 5508d3f2 >2012-12-17 08:41:09,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] FINISH, HSMClearTaskVDSCommand, log id: 5508d3f2 >2012-12-17 08:41:09,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [41c296ff] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3692e07b, log id: 48b82caa >2012-12-17 08:41:09,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:09,921 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:41:09,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2d23e813 >2012-12-17 08:41:09,946 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [41c296ff] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:41:10,034 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [6c95e244] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:41:10,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 16f2d2bb >2012-12-17 08:41:10,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e9038cc >2012-12-17 08:41:10,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:10,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, SpmStopVDSCommand, log id: 1e9038cc >2012-12-17 08:41:10,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, ResetIrsVDSCommand, log id: 16f2d2bb >2012-12-17 08:41:10,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9efdf6b >2012-12-17 08:41:10,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:10,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [6c95e244] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9efdf6b >2012-12-17 08:41:11,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:11,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4618 >2012-12-17 08:41:11,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4618, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63a20d01 >2012-12-17 08:41:11,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 112a9648-b185-44ca-a2b8-349fc0d0bafa >2012-12-17 08:41:12,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 112a9648-b185-44ca-a2b8-349fc0d0bafa task status = finished >2012-12-17 08:41:12,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM >2012-12-17 08:41:12,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=112a9648-b185-44ca-a2b8-349fc0d0bafa), log id: 5e05cf34 >2012-12-17 08:41:12,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 5e05cf34 >2012-12-17 08:41:12,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f668f25, log id: 63a20d01 >2012-12-17 08:41:12,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:12,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:12,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20465ca >2012-12-17 08:41:12,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20465ca >2012-12-17 08:41:12,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2d23e813 >2012-12-17 08:41:12,258 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:12,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6eb6b1ce >2012-12-17 08:41:12,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:12,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d3639a1 >2012-12-17 08:41:12,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d3639a1 >2012-12-17 08:41:12,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6eb6b1ce >2012-12-17 08:41:12,279 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [181d86f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:17,089 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:41:21,325 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [48fde2a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:41:21,606 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [28e17066] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:41:21,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20dbea2c >2012-12-17 08:41:21,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45724812 >2012-12-17 08:41:21,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:21,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, SpmStopVDSCommand, log id: 45724812 >2012-12-17 08:41:21,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, ResetIrsVDSCommand, log id: 20dbea2c >2012-12-17 08:41:21,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [28e17066] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bbad73b >2012-12-17 08:41:21,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [28e17066] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:21,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [28e17066] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bbad73b >2012-12-17 08:41:27,275 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:41:32,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:32,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4619 >2012-12-17 08:41:32,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4619, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38f4581d >2012-12-17 08:41:32,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling started: taskId = 9e744fc2-fb9e-4b53-be59-d8db23236c4f >2012-12-17 08:41:33,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling ended: taskId = 9e744fc2-fb9e-4b53-be59-d8db23236c4f task status = finished >2012-12-17 08:41:33,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] spmStart polling ended, spm status: SPM >2012-12-17 08:41:33,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9e744fc2-fb9e-4b53-be59-d8db23236c4f), log id: a079d55 >2012-12-17 08:41:33,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, HSMClearTaskVDSCommand, log id: a079d55 >2012-12-17 08:41:33,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [50db5e16] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@79ea3c97, log id: 38f4581d >2012-12-17 08:41:33,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:33,176 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:41:33,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3345386a >2012-12-17 08:41:33,194 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [50db5e16] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:41:33,423 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [51589818] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:41:33,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21df1f84 >2012-12-17 08:41:33,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2ba9a65f >2012-12-17 08:41:33,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:33,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, SpmStopVDSCommand, log id: 2ba9a65f >2012-12-17 08:41:33,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, ResetIrsVDSCommand, log id: 21df1f84 >2012-12-17 08:41:33,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [51589818] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39c103a5 >2012-12-17 08:41:33,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:33,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [51589818] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39c103a5 >2012-12-17 08:41:33,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:33,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4620 >2012-12-17 08:41:33,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4620, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 368f4227 >2012-12-17 08:41:33,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = abfe3d5b-6471-49bf-93fa-5e545f014038 >2012-12-17 08:41:34,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = abfe3d5b-6471-49bf-93fa-5e545f014038 task status = finished >2012-12-17 08:41:34,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM >2012-12-17 08:41:34,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=abfe3d5b-6471-49bf-93fa-5e545f014038), log id: 44803a1a >2012-12-17 08:41:34,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 44803a1a >2012-12-17 08:41:34,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e66a418, log id: 368f4227 >2012-12-17 08:41:34,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:34,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:34,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d327944 >2012-12-17 08:41:34,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d327944 >2012-12-17 08:41:34,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3345386a >2012-12-17 08:41:34,989 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:34,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32e24cbd >2012-12-17 08:41:34,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:34,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79b72005 >2012-12-17 08:41:35,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79b72005 >2012-12-17 08:41:35,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32e24cbd >2012-12-17 08:41:35,009 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:37,468 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:41:43,831 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [95f5063] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:41:43,961 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [6bb29080] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:41:44,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 70470419 >2012-12-17 08:41:44,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e2409aa >2012-12-17 08:41:44,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:44,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStopVDSCommand, log id: 1e2409aa >2012-12-17 08:41:44,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, ResetIrsVDSCommand, log id: 70470419 >2012-12-17 08:41:44,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3fa0fff0 >2012-12-17 08:41:44,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:44,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3fa0fff0 >2012-12-17 08:41:47,703 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:41:54,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:54,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4621 >2012-12-17 08:41:54,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4621, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d457424 >2012-12-17 08:41:54,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling started: taskId = a44e1167-47d2-4270-868f-f49c5f62a28b >2012-12-17 08:41:55,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended: taskId = a44e1167-47d2-4270-868f-f49c5f62a28b task status = finished >2012-12-17 08:41:55,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] spmStart polling ended, spm status: SPM >2012-12-17 08:41:55,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a44e1167-47d2-4270-868f-f49c5f62a28b), log id: 7193a3ac >2012-12-17 08:41:55,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, HSMClearTaskVDSCommand, log id: 7193a3ac >2012-12-17 08:41:55,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [51f97e37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13deab75, log id: 2d457424 >2012-12-17 08:41:55,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:56,023 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:41:56,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42a64369 >2012-12-17 08:41:56,055 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [51f97e37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:41:56,197 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [1d8c274e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:41:56,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65fff724 >2012-12-17 08:41:56,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 799b3086 >2012-12-17 08:41:56,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:56,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStopVDSCommand, log id: 799b3086 >2012-12-17 08:41:56,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, ResetIrsVDSCommand, log id: 65fff724 >2012-12-17 08:41:56,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6319a1a5 >2012-12-17 08:41:56,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:56,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6319a1a5 >2012-12-17 08:41:56,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:41:56,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4622 >2012-12-17 08:41:56,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4622, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 582723d1 >2012-12-17 08:41:56,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling started: taskId = 485a87ec-e214-41fe-9006-020eb5d3cab0 >2012-12-17 08:41:57,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling ended: taskId = 485a87ec-e214-41fe-9006-020eb5d3cab0 task status = finished >2012-12-17 08:41:57,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] spmStart polling ended, spm status: SPM >2012-12-17 08:41:57,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=485a87ec-e214-41fe-9006-020eb5d3cab0), log id: 408c1872 >2012-12-17 08:41:57,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMClearTaskVDSCommand, log id: 408c1872 >2012-12-17 08:41:57,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d990579, log id: 582723d1 >2012-12-17 08:41:57,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:41:57,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:57,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63435e95 >2012-12-17 08:41:57,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63435e95 >2012-12-17 08:41:57,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42a64369 >2012-12-17 08:41:57,690 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:57,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5bf6bb1c >2012-12-17 08:41:57,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:41:57,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ae2eb8e >2012-12-17 08:41:57,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ae2eb8e >2012-12-17 08:41:57,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [2f94bcb4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5bf6bb1c >2012-12-17 08:41:57,711 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [2f94bcb4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:41:57,942 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:42:06,610 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [406132e6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:42:06,861 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [6acb626c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:42:06,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a4f4afc >2012-12-17 08:42:06,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 797b5b5a >2012-12-17 08:42:06,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:07,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, SpmStopVDSCommand, log id: 797b5b5a >2012-12-17 08:42:07,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, ResetIrsVDSCommand, log id: 5a4f4afc >2012-12-17 08:42:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 118304c7 >2012-12-17 08:42:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [6acb626c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [6acb626c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 118304c7 >2012-12-17 08:42:08,129 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:42:17,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:42:17,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4623 >2012-12-17 08:42:17,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4623, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4237ed0f >2012-12-17 08:42:17,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling started: taskId = e8b915c4-d475-4737-80b6-0f1f78ad2a55 >2012-12-17 08:42:18,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling ended: taskId = e8b915c4-d475-4737-80b6-0f1f78ad2a55 task status = finished >2012-12-17 08:42:18,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] spmStart polling ended, spm status: SPM >2012-12-17 08:42:18,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e8b915c4-d475-4737-80b6-0f1f78ad2a55), log id: 19ea12c4 >2012-12-17 08:42:18,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] FINISH, HSMClearTaskVDSCommand, log id: 19ea12c4 >2012-12-17 08:42:18,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [2cd85c0e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56760f26, log id: 4237ed0f >2012-12-17 08:42:18,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:42:18,445 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:42:18,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5016d94d >2012-12-17 08:42:18,467 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2cd85c0e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:42:18,702 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5f1742db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:42:18,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e95e1e3 >2012-12-17 08:42:18,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 30f90f68 >2012-12-17 08:42:18,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:18,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, SpmStopVDSCommand, log id: 30f90f68 >2012-12-17 08:42:18,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, ResetIrsVDSCommand, log id: 2e95e1e3 >2012-12-17 08:42:18,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c63e19c >2012-12-17 08:42:18,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:18,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c63e19c >2012-12-17 08:42:18,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:42:19,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4624 >2012-12-17 08:42:19,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4624, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10a1d5c4 >2012-12-17 08:42:19,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling started: taskId = 41f26e2a-9d26-4f4e-8a83-c0f643520a84 >2012-12-17 08:42:20,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended: taskId = 41f26e2a-9d26-4f4e-8a83-c0f643520a84 task status = finished >2012-12-17 08:42:20,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended, spm status: SPM >2012-12-17 08:42:20,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=41f26e2a-9d26-4f4e-8a83-c0f643520a84), log id: 7bf9e7d6 >2012-12-17 08:42:20,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMClearTaskVDSCommand, log id: 7bf9e7d6 >2012-12-17 08:42:20,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53e3d230, log id: 10a1d5c4 >2012-12-17 08:42:20,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:42:20,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:20,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11f4d41b >2012-12-17 08:42:20,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11f4d41b >2012-12-17 08:42:20,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5016d94d >2012-12-17 08:42:20,146 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:42:20,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 60787385 >2012-12-17 08:42:20,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:20,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 46b705b6 >2012-12-17 08:42:20,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 46b705b6 >2012-12-17 08:42:20,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 60787385 >2012-12-17 08:42:20,167 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:42:28,521 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:42:29,034 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [a2fcda8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:42:29,161 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [27478f27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:42:29,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 19ad8938 >2012-12-17 08:42:29,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 335c39ca >2012-12-17 08:42:29,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:29,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, SpmStopVDSCommand, log id: 335c39ca >2012-12-17 08:42:29,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, ResetIrsVDSCommand, log id: 19ad8938 >2012-12-17 08:42:29,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d4ce839 >2012-12-17 08:42:29,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:29,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d4ce839 >2012-12-17 08:42:38,710 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:42:39,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:42:39,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4625 >2012-12-17 08:42:39,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4625, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a7974bc >2012-12-17 08:42:39,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling started: taskId = 926350d7-83d9-4e68-9868-3f2732042968 >2012-12-17 08:42:40,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended: taskId = 926350d7-83d9-4e68-9868-3f2732042968 task status = finished >2012-12-17 08:42:40,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] spmStart polling ended, spm status: SPM >2012-12-17 08:42:40,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=926350d7-83d9-4e68-9868-3f2732042968), log id: 58587d6b >2012-12-17 08:42:40,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, HSMClearTaskVDSCommand, log id: 58587d6b >2012-12-17 08:42:40,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [7543fd1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1992291d, log id: 6a7974bc >2012-12-17 08:42:40,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:42:41,078 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:42:41,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 478d8b67 >2012-12-17 08:42:41,106 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7543fd1a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:42:41,214 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [e4c8c37] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:42:41,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f021e29 >2012-12-17 08:42:41,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d1a071e >2012-12-17 08:42:41,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:41,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStopVDSCommand, log id: 4d1a071e >2012-12-17 08:42:41,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, ResetIrsVDSCommand, log id: 5f021e29 >2012-12-17 08:42:41,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24202844 >2012-12-17 08:42:41,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:41,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24202844 >2012-12-17 08:42:41,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:42:41,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4626 >2012-12-17 08:42:41,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4626, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77c97917 >2012-12-17 08:42:41,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling started: taskId = 1e173aa2-fd41-4a03-873f-8401c8333a5f >2012-12-17 08:42:42,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling ended: taskId = 1e173aa2-fd41-4a03-873f-8401c8333a5f task status = finished >2012-12-17 08:42:42,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] spmStart polling ended, spm status: SPM >2012-12-17 08:42:42,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e173aa2-fd41-4a03-873f-8401c8333a5f), log id: 466f3d5c >2012-12-17 08:42:42,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMClearTaskVDSCommand, log id: 466f3d5c >2012-12-17 08:42:42,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31258724, log id: 77c97917 >2012-12-17 08:42:42,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:42:42,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:42,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d5dd288 >2012-12-17 08:42:42,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d5dd288 >2012-12-17 08:42:42,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 478d8b67 >2012-12-17 08:42:42,931 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:42:42,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30fe0033 >2012-12-17 08:42:42,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:42,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3287c588 >2012-12-17 08:42:42,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3287c588 >2012-12-17 08:42:42,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [41698d74] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30fe0033 >2012-12-17 08:42:42,951 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [41698d74] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:42:48,916 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:42:51,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [25db9572] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:42:51,936 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [34599a45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:42:52,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 19ffdefa >2012-12-17 08:42:52,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b547485 >2012-12-17 08:42:52,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:52,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, SpmStopVDSCommand, log id: 4b547485 >2012-12-17 08:42:52,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, ResetIrsVDSCommand, log id: 19ffdefa >2012-12-17 08:42:52,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d5070bb >2012-12-17 08:42:52,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:42:52,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d5070bb >2012-12-17 08:42:59,171 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:43:02,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:02,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4627 >2012-12-17 08:43:02,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4627, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f548a68 >2012-12-17 08:43:02,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling started: taskId = f9dba6c0-56c0-4914-aa0a-1a8b584713a9 >2012-12-17 08:43:03,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling ended: taskId = f9dba6c0-56c0-4914-aa0a-1a8b584713a9 task status = finished >2012-12-17 08:43:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] spmStart polling ended, spm status: SPM >2012-12-17 08:43:03,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9dba6c0-56c0-4914-aa0a-1a8b584713a9), log id: 3075ad3c >2012-12-17 08:43:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, HSMClearTaskVDSCommand, log id: 3075ad3c >2012-12-17 08:43:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [5fbba580] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7dc056fe, log id: 5f548a68 >2012-12-17 08:43:03,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:03,580 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:43:03,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29025348 >2012-12-17 08:43:03,626 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5fbba580] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:03,750 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [30867547] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:03,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b39a81a >2012-12-17 08:43:03,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ac6f7b6 >2012-12-17 08:43:03,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:03,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, SpmStopVDSCommand, log id: 3ac6f7b6 >2012-12-17 08:43:03,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, ResetIrsVDSCommand, log id: 5b39a81a >2012-12-17 08:43:03,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ef2f7d >2012-12-17 08:43:03,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:03,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ef2f7d >2012-12-17 08:43:04,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:04,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4628 >2012-12-17 08:43:04,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4628, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ccb66f7 >2012-12-17 08:43:04,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling started: taskId = 0bcdc6fe-1fa5-40cc-a25c-055de6025850 >2012-12-17 08:43:05,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended: taskId = 0bcdc6fe-1fa5-40cc-a25c-055de6025850 task status = finished >2012-12-17 08:43:05,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] spmStart polling ended, spm status: SPM >2012-12-17 08:43:05,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0bcdc6fe-1fa5-40cc-a25c-055de6025850), log id: 6ad96cf9 >2012-12-17 08:43:05,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMClearTaskVDSCommand, log id: 6ad96cf9 >2012-12-17 08:43:05,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6881863c, log id: 6ccb66f7 >2012-12-17 08:43:05,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:05,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:05,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d24c978 >2012-12-17 08:43:05,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d24c978 >2012-12-17 08:43:05,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29025348 >2012-12-17 08:43:05,217 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:05,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a26f43a >2012-12-17 08:43:05,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:05,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51b5fa83 >2012-12-17 08:43:05,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51b5fa83 >2012-12-17 08:43:05,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [419474a0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a26f43a >2012-12-17 08:43:05,252 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [419474a0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:09,387 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:43:14,103 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [42bc2eb6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:14,227 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [49c4851d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:14,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20b0ed13 >2012-12-17 08:43:14,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5db5d8df >2012-12-17 08:43:14,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:14,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, SpmStopVDSCommand, log id: 5db5d8df >2012-12-17 08:43:14,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, ResetIrsVDSCommand, log id: 20b0ed13 >2012-12-17 08:43:14,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b741b19 >2012-12-17 08:43:14,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:14,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b741b19 >2012-12-17 08:43:19,594 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:43:24,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:24,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4629 >2012-12-17 08:43:24,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4629, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1dff7354 >2012-12-17 08:43:24,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling started: taskId = b6cff0ee-6f1f-4124-ad17-1bb770f85473 >2012-12-17 08:43:25,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling ended: taskId = b6cff0ee-6f1f-4124-ad17-1bb770f85473 task status = finished >2012-12-17 08:43:25,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] spmStart polling ended, spm status: SPM >2012-12-17 08:43:25,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b6cff0ee-6f1f-4124-ad17-1bb770f85473), log id: 1b3876c1 >2012-12-17 08:43:25,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, HSMClearTaskVDSCommand, log id: 1b3876c1 >2012-12-17 08:43:25,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [653dcf69] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17fc86dd, log id: 1dff7354 >2012-12-17 08:43:26,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:26,077 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:43:26,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21707aaa >2012-12-17 08:43:26,098 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [653dcf69] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:26,212 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [3eec3b83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:26,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50bdc77c >2012-12-17 08:43:26,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7eb94769 >2012-12-17 08:43:26,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:26,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, SpmStopVDSCommand, log id: 7eb94769 >2012-12-17 08:43:26,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, ResetIrsVDSCommand, log id: 50bdc77c >2012-12-17 08:43:26,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bf60e71 >2012-12-17 08:43:26,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:26,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bf60e71 >2012-12-17 08:43:26,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:26,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4630 >2012-12-17 08:43:26,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4630, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c74ee7f >2012-12-17 08:43:26,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = b26927ca-3904-42e4-b775-f99a5105914f >2012-12-17 08:43:27,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = b26927ca-3904-42e4-b775-f99a5105914f task status = finished >2012-12-17 08:43:27,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM >2012-12-17 08:43:27,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b26927ca-3904-42e4-b775-f99a5105914f), log id: 4f39fc82 >2012-12-17 08:43:27,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: 4f39fc82 >2012-12-17 08:43:27,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67f0e370, log id: c74ee7f >2012-12-17 08:43:27,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:27,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:27,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5ff3cf0c >2012-12-17 08:43:27,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5ff3cf0c >2012-12-17 08:43:27,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21707aaa >2012-12-17 08:43:27,769 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:27,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f9e4e0a >2012-12-17 08:43:27,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:27,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: a4ef26f >2012-12-17 08:43:27,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: a4ef26f >2012-12-17 08:43:27,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f9e4e0a >2012-12-17 08:43:27,791 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:29,858 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:43:36,658 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [778f4477] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:36,818 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [41dfde4b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:37,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1bd9ac11 >2012-12-17 08:43:37,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 91402ba >2012-12-17 08:43:37,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:37,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, SpmStopVDSCommand, log id: 91402ba >2012-12-17 08:43:37,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, ResetIrsVDSCommand, log id: 1bd9ac11 >2012-12-17 08:43:37,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1864106e >2012-12-17 08:43:37,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:37,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1864106e >2012-12-17 08:43:40,070 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:43:47,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:47,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4631 >2012-12-17 08:43:47,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4631, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c14c7b1 >2012-12-17 08:43:47,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling started: taskId = a8d01c5a-dcb7-4094-8858-771fd6a48f88 >2012-12-17 08:43:48,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended: taskId = a8d01c5a-dcb7-4094-8858-771fd6a48f88 task status = finished >2012-12-17 08:43:48,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] spmStart polling ended, spm status: SPM >2012-12-17 08:43:48,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a8d01c5a-dcb7-4094-8858-771fd6a48f88), log id: 427b5e80 >2012-12-17 08:43:48,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, HSMClearTaskVDSCommand, log id: 427b5e80 >2012-12-17 08:43:48,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [5a89a26c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31794799, log id: 5c14c7b1 >2012-12-17 08:43:48,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:48,509 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:43:48,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79fbfef >2012-12-17 08:43:48,534 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5a89a26c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:48,688 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [5ad4c70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:48,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f28bffc >2012-12-17 08:43:48,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3010976e >2012-12-17 08:43:48,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:49,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, SpmStopVDSCommand, log id: 3010976e >2012-12-17 08:43:49,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, ResetIrsVDSCommand, log id: 5f28bffc >2012-12-17 08:43:49,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c85be4e >2012-12-17 08:43:49,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5ad4c70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:49,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5ad4c70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c85be4e >2012-12-17 08:43:49,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:43:49,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4632 >2012-12-17 08:43:49,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4632, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20c10828 >2012-12-17 08:43:49,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling started: taskId = 85785197-903d-43d8-9ec7-a9980e5fd689 >2012-12-17 08:43:51,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling ended: taskId = 85785197-903d-43d8-9ec7-a9980e5fd689 task status = finished >2012-12-17 08:43:51,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] spmStart polling ended, spm status: SPM >2012-12-17 08:43:51,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=85785197-903d-43d8-9ec7-a9980e5fd689), log id: 338d1d56 >2012-12-17 08:43:51,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMClearTaskVDSCommand, log id: 338d1d56 >2012-12-17 08:43:51,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71af44c5, log id: 20c10828 >2012-12-17 08:43:51,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:43:51,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:51,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16ce6d05 >2012-12-17 08:43:51,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16ce6d05 >2012-12-17 08:43:51,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79fbfef >2012-12-17 08:43:51,368 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:51,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a576ef5 >2012-12-17 08:43:51,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:43:51,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ed06cd7 >2012-12-17 08:43:51,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ed06cd7 >2012-12-17 08:43:51,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [2649fcd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a576ef5 >2012-12-17 08:43:51,389 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [2649fcd5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:43:59,598 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [16c2b2c4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:43:59,736 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [40a5fc59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:43:59,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e0adfd >2012-12-17 08:43:59,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5440ad2a >2012-12-17 08:43:59,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SpmStopVDSCommand, log id: 5440ad2a >2012-12-17 08:44:00,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, ResetIrsVDSCommand, log id: 2e0adfd >2012-12-17 08:44:00,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: efbf806 >2012-12-17 08:44:00,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:00,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: efbf806 >2012-12-17 08:44:01,221 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:44:10,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:10,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4633 >2012-12-17 08:44:10,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4633, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 25316b5e >2012-12-17 08:44:10,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling started: taskId = 16704ac1-74c8-4e6c-b523-11d760e37cf5 >2012-12-17 08:44:11,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling ended: taskId = 16704ac1-74c8-4e6c-b523-11d760e37cf5 task status = finished >2012-12-17 08:44:11,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] spmStart polling ended, spm status: SPM >2012-12-17 08:44:11,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=16704ac1-74c8-4e6c-b523-11d760e37cf5), log id: b786629 >2012-12-17 08:44:11,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, HSMClearTaskVDSCommand, log id: b786629 >2012-12-17 08:44:11,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [1daf49b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c291a4d, log id: 25316b5e >2012-12-17 08:44:11,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:11,699 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:44:11,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 787e9bf5 >2012-12-17 08:44:11,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1daf49b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:44:11,836 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [61acfe5d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:44:11,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ee48e16 >2012-12-17 08:44:11,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14d8d12d >2012-12-17 08:44:11,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:12,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, SpmStopVDSCommand, log id: 14d8d12d >2012-12-17 08:44:12,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, ResetIrsVDSCommand, log id: 5ee48e16 >2012-12-17 08:44:12,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ca04d2a >2012-12-17 08:44:12,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:12,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ca04d2a >2012-12-17 08:44:12,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:12,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4634 >2012-12-17 08:44:12,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4634, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bd20f88 >2012-12-17 08:44:12,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling started: taskId = c4a4c340-eeaf-4874-8e45-c62172294337 >2012-12-17 08:44:13,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended: taskId = c4a4c340-eeaf-4874-8e45-c62172294337 task status = finished >2012-12-17 08:44:13,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] spmStart polling ended, spm status: SPM >2012-12-17 08:44:13,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c4a4c340-eeaf-4874-8e45-c62172294337), log id: 6c334f1b >2012-12-17 08:44:13,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMClearTaskVDSCommand, log id: 6c334f1b >2012-12-17 08:44:13,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c8ead1c, log id: bd20f88 >2012-12-17 08:44:13,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:13,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:13,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27090873 >2012-12-17 08:44:13,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27090873 >2012-12-17 08:44:13,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 787e9bf5 >2012-12-17 08:44:13,442 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:44:13,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 740bd1e2 >2012-12-17 08:44:13,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:13,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76ab3960 >2012-12-17 08:44:13,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76ab3960 >2012-12-17 08:44:13,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [566819d6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 740bd1e2 >2012-12-17 08:44:13,472 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [566819d6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:44:21,609 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:44:22,244 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [1cd1ecc2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:44:22,432 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [107aac36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:44:22,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4ac66000 >2012-12-17 08:44:22,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 291a661b >2012-12-17 08:44:22,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:22,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SpmStopVDSCommand, log id: 291a661b >2012-12-17 08:44:22,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, ResetIrsVDSCommand, log id: 4ac66000 >2012-12-17 08:44:22,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d71b561 >2012-12-17 08:44:22,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:22,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d71b561 >2012-12-17 08:44:31,880 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:44:32,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:32,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4635 >2012-12-17 08:44:32,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4635, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 480be024 >2012-12-17 08:44:32,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling started: taskId = efa79c84-0c93-4e9e-9125-3b9b4731c98e >2012-12-17 08:44:33,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling ended: taskId = efa79c84-0c93-4e9e-9125-3b9b4731c98e task status = finished >2012-12-17 08:44:33,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] spmStart polling ended, spm status: SPM >2012-12-17 08:44:33,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=efa79c84-0c93-4e9e-9125-3b9b4731c98e), log id: 4d33545f >2012-12-17 08:44:33,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, HSMClearTaskVDSCommand, log id: 4d33545f >2012-12-17 08:44:33,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [24452d4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e85091, log id: 480be024 >2012-12-17 08:44:33,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:33,934 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:44:33,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cac3416 >2012-12-17 08:44:33,963 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [24452d4e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:44:34,147 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [7ad32f3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:44:34,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4da16536 >2012-12-17 08:44:34,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e8a8494 >2012-12-17 08:44:34,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:34,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, SpmStopVDSCommand, log id: 6e8a8494 >2012-12-17 08:44:34,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, ResetIrsVDSCommand, log id: 4da16536 >2012-12-17 08:44:34,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3de8a623 >2012-12-17 08:44:34,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7ad32f3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:34,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7ad32f3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3de8a623 >2012-12-17 08:44:34,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:34,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4636 >2012-12-17 08:44:34,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4636, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 188edafe >2012-12-17 08:44:34,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = 9fdad1fc-6c4c-4e13-a8d5-4dccc5354918 >2012-12-17 08:44:35,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = 9fdad1fc-6c4c-4e13-a8d5-4dccc5354918 task status = finished >2012-12-17 08:44:35,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM >2012-12-17 08:44:35,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9fdad1fc-6c4c-4e13-a8d5-4dccc5354918), log id: 4833c79c >2012-12-17 08:44:35,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 4833c79c >2012-12-17 08:44:35,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f75f7cc, log id: 188edafe >2012-12-17 08:44:36,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:36,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:36,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dd9a1c >2012-12-17 08:44:36,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dd9a1c >2012-12-17 08:44:36,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cac3416 >2012-12-17 08:44:36,375 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:44:36,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54b0a4b5 >2012-12-17 08:44:36,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:36,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 510bfc78 >2012-12-17 08:44:36,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 510bfc78 >2012-12-17 08:44:36,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54b0a4b5 >2012-12-17 08:44:36,395 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [55d82264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:44:42,052 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:44:44,509 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [51589818] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:44:44,641 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [3f9576fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:44:44,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45a15c4f >2012-12-17 08:44:44,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bc46a7d >2012-12-17 08:44:44,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:45,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, SpmStopVDSCommand, log id: 2bc46a7d >2012-12-17 08:44:45,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, ResetIrsVDSCommand, log id: 45a15c4f >2012-12-17 08:44:45,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47f9392a >2012-12-17 08:44:45,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:45,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47f9392a >2012-12-17 08:44:52,230 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:44:56,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:56,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4637 >2012-12-17 08:44:56,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4637, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 261f56a3 >2012-12-17 08:44:56,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling started: taskId = 4d8337f7-37e5-4efa-9fa4-ce887e8f4525 >2012-12-17 08:44:57,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling ended: taskId = 4d8337f7-37e5-4efa-9fa4-ce887e8f4525 task status = finished >2012-12-17 08:44:57,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] spmStart polling ended, spm status: SPM >2012-12-17 08:44:57,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [f91baee] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d8337f7-37e5-4efa-9fa4-ce887e8f4525), log id: 7e1e5c8d >2012-12-17 08:44:57,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, HSMClearTaskVDSCommand, log id: 7e1e5c8d >2012-12-17 08:44:57,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [f91baee] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b4e6daa, log id: 261f56a3 >2012-12-17 08:44:57,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:57,512 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:44:57,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6dc881f0 >2012-12-17 08:44:57,531 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [f91baee] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:44:57,692 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [3d206f3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:44:57,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 303f287c >2012-12-17 08:44:57,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7499f28 >2012-12-17 08:44:57,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:58,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SpmStopVDSCommand, log id: 7499f28 >2012-12-17 08:44:58,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, ResetIrsVDSCommand, log id: 303f287c >2012-12-17 08:44:58,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e28a95 >2012-12-17 08:44:58,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:58,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e28a95 >2012-12-17 08:44:58,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:44:58,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4638 >2012-12-17 08:44:58,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4638, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c49fe60 >2012-12-17 08:44:58,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 0e4bcbb4-c94b-474d-9857-527c70a4e7c3 >2012-12-17 08:44:59,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 0e4bcbb4-c94b-474d-9857-527c70a4e7c3 task status = finished >2012-12-17 08:44:59,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM >2012-12-17 08:44:59,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0e4bcbb4-c94b-474d-9857-527c70a4e7c3), log id: 631218b6 >2012-12-17 08:44:59,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 631218b6 >2012-12-17 08:44:59,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3914878a, log id: 7c49fe60 >2012-12-17 08:44:59,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:44:59,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:59,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 703581d5 >2012-12-17 08:44:59,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 703581d5 >2012-12-17 08:44:59,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6dc881f0 >2012-12-17 08:44:59,354 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:44:59,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2899f8e0 >2012-12-17 08:44:59,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:44:59,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 345d0a0e >2012-12-17 08:44:59,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 345d0a0e >2012-12-17 08:44:59,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2899f8e0 >2012-12-17 08:44:59,374 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Checking autorecoverable hosts >2012-12-17 08:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Autorecovering 0 hosts >2012-12-17 08:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Checking autorecoverable hosts done >2012-12-17 08:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1ab078c7] Autorecovering storage domains is disabled, skipping >2012-12-17 08:45:02,462 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:45:08,253 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [b1109c1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:45:08,388 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [58b8166b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:45:08,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14e1fa40 >2012-12-17 08:45:08,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21b88f07 >2012-12-17 08:45:08,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:08,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, SpmStopVDSCommand, log id: 21b88f07 >2012-12-17 08:45:08,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, ResetIrsVDSCommand, log id: 14e1fa40 >2012-12-17 08:45:08,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d0e6b55 >2012-12-17 08:45:08,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:08,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d0e6b55 >2012-12-17 08:45:12,704 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:45:18,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:45:18,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4639 >2012-12-17 08:45:18,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4639, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d8080c >2012-12-17 08:45:18,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling started: taskId = e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd >2012-12-17 08:45:19,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling ended: taskId = e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd task status = finished >2012-12-17 08:45:19,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] spmStart polling ended, spm status: SPM >2012-12-17 08:45:19,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e46086e4-41fb-4dd4-9fd5-6ca8e2b86ebd), log id: 7b831ec9 >2012-12-17 08:45:19,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, HSMClearTaskVDSCommand, log id: 7b831ec9 >2012-12-17 08:45:19,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [5863fdac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5a5d78, log id: 32d8080c >2012-12-17 08:45:19,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:45:19,963 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:45:19,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74d05267 >2012-12-17 08:45:19,981 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [5863fdac] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:45:20,079 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:45:20,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3c4f4da5 >2012-12-17 08:45:20,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1125beb1 >2012-12-17 08:45:20,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:20,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SpmStopVDSCommand, log id: 1125beb1 >2012-12-17 08:45:21,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, ResetIrsVDSCommand, log id: 3c4f4da5 >2012-12-17 08:45:21,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49ba87df >2012-12-17 08:45:21,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:21,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49ba87df >2012-12-17 08:45:21,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:45:21,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4640 >2012-12-17 08:45:21,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4640, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 90c5b1b >2012-12-17 08:45:21,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = c012a391-1710-40e3-8b67-41dec16418d5 >2012-12-17 08:45:22,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = c012a391-1710-40e3-8b67-41dec16418d5 task status = finished >2012-12-17 08:45:22,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM >2012-12-17 08:45:22,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c012a391-1710-40e3-8b67-41dec16418d5), log id: 78a5ee45 >2012-12-17 08:45:22,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 78a5ee45 >2012-12-17 08:45:22,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55be31e4, log id: 90c5b1b >2012-12-17 08:45:22,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:45:22,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:22,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67eb7667 >2012-12-17 08:45:22,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67eb7667 >2012-12-17 08:45:22,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74d05267 >2012-12-17 08:45:22,498 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:22,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 25432948 >2012-12-17 08:45:22,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:22,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d78401e >2012-12-17 08:45:22,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d78401e >2012-12-17 08:45:22,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 25432948 >2012-12-17 08:45:22,519 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:22,893 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:45:31,382 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65a08bc7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:45:31,548 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [786fe4b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:45:31,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 72c567eb >2012-12-17 08:45:31,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15aa1c0 >2012-12-17 08:45:31,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:31,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SpmStopVDSCommand, log id: 15aa1c0 >2012-12-17 08:45:31,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, ResetIrsVDSCommand, log id: 72c567eb >2012-12-17 08:45:31,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1118a1cc >2012-12-17 08:45:31,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:31,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1118a1cc >2012-12-17 08:45:33,120 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:45:41,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:45:41,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4641 >2012-12-17 08:45:41,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4641, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5928f331 >2012-12-17 08:45:41,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling started: taskId = 2b6bcbb5-e205-4e06-8bb2-1bf009f5da27 >2012-12-17 08:45:42,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended: taskId = 2b6bcbb5-e205-4e06-8bb2-1bf009f5da27 task status = finished >2012-12-17 08:45:43,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] spmStart polling ended, spm status: SPM >2012-12-17 08:45:43,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b6bcbb5-e205-4e06-8bb2-1bf009f5da27), log id: 6339ab12 >2012-12-17 08:45:43,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, HSMClearTaskVDSCommand, log id: 6339ab12 >2012-12-17 08:45:43,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [3b972285] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f0e5da2, log id: 5928f331 >2012-12-17 08:45:43,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:45:43,082 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:45:43,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1dd5371 >2012-12-17 08:45:43,101 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [3b972285] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:45:43,182 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [536655ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:45:43,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6ab7477a >2012-12-17 08:45:43,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e4789f1 >2012-12-17 08:45:43,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:43,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:45:43,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, SpmStopVDSCommand, log id: e4789f1 >2012-12-17 08:45:43,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, ResetIrsVDSCommand, log id: 6ab7477a >2012-12-17 08:45:43,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [536655ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d0df750 >2012-12-17 08:45:43,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:43,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [536655ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d0df750 >2012-12-17 08:45:43,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:45:43,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4642 >2012-12-17 08:45:43,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4642, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f0027a6 >2012-12-17 08:45:43,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = 215685ad-0f84-4099-8b04-4885cfe519cb >2012-12-17 08:45:44,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = 215685ad-0f84-4099-8b04-4885cfe519cb task status = finished >2012-12-17 08:45:44,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM >2012-12-17 08:45:44,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=215685ad-0f84-4099-8b04-4885cfe519cb), log id: 366a31d0 >2012-12-17 08:45:44,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 366a31d0 >2012-12-17 08:45:44,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75ec6a98, log id: 6f0027a6 >2012-12-17 08:45:44,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:45:44,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:44,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5385f790 >2012-12-17 08:45:44,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5385f790 >2012-12-17 08:45:44,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1dd5371 >2012-12-17 08:45:44,882 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:44,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 789137f1 >2012-12-17 08:45:44,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:44,883 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:45:44,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d0f43ab >2012-12-17 08:45:44,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: aefec30 >2012-12-17 08:45:44,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: aefec30 >2012-12-17 08:45:44,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 789137f1 >2012-12-17 08:45:44,902 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:44,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:44,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2616771b >2012-12-17 08:45:44,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2616771b >2012-12-17 08:45:44,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d0f43ab >2012-12-17 08:45:44,922 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:44,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: ce5a126 >2012-12-17 08:45:44,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:44,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10572dbf >2012-12-17 08:45:44,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10572dbf >2012-12-17 08:45:44,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: ce5a126 >2012-12-17 08:45:44,942 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:45:53,761 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [41698d74] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:45:54,094 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [6cc0b445] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:45:54,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7dd34308 >2012-12-17 08:45:54,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41de21e1 >2012-12-17 08:45:54,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:54,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SpmStopVDSCommand, log id: 41de21e1 >2012-12-17 08:45:54,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, ResetIrsVDSCommand, log id: 7dd34308 >2012-12-17 08:45:54,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18dcc6ea >2012-12-17 08:45:54,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:45:54,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18dcc6ea >2012-12-17 08:46:03,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:46:04,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:04,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4643 >2012-12-17 08:46:04,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4643, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 764f82e5 >2012-12-17 08:46:04,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling started: taskId = 3597ab48-d31a-40cb-ae91-fb17bc14740e >2012-12-17 08:46:05,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling ended: taskId = 3597ab48-d31a-40cb-ae91-fb17bc14740e task status = finished >2012-12-17 08:46:05,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] spmStart polling ended, spm status: SPM >2012-12-17 08:46:05,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [56986c52] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3597ab48-d31a-40cb-ae91-fb17bc14740e), log id: c308e8c >2012-12-17 08:46:05,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, HSMClearTaskVDSCommand, log id: c308e8c >2012-12-17 08:46:05,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [56986c52] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47de8f94, log id: 764f82e5 >2012-12-17 08:46:05,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:06,106 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:46:06,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d0449c >2012-12-17 08:46:06,116 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [56986c52] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:46:06,316 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:46:06,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7b00e96b >2012-12-17 08:46:06,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26ec42f0 >2012-12-17 08:46:06,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:06,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, SpmStopVDSCommand, log id: 26ec42f0 >2012-12-17 08:46:06,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, ResetIrsVDSCommand, log id: 7b00e96b >2012-12-17 08:46:06,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25d0f66c >2012-12-17 08:46:06,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:06,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25d0f66c >2012-12-17 08:46:06,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:06,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4644 >2012-12-17 08:46:06,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4644, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 485ba2b2 >2012-12-17 08:46:06,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = 684d6c9d-2b10-4131-a407-1ab0bb9f81df >2012-12-17 08:46:07,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = 684d6c9d-2b10-4131-a407-1ab0bb9f81df task status = finished >2012-12-17 08:46:07,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM >2012-12-17 08:46:07,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=684d6c9d-2b10-4131-a407-1ab0bb9f81df), log id: e57fd24 >2012-12-17 08:46:07,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: e57fd24 >2012-12-17 08:46:07,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@32ac4dd7, log id: 485ba2b2 >2012-12-17 08:46:07,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:07,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:07,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f5ba3b9 >2012-12-17 08:46:07,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f5ba3b9 >2012-12-17 08:46:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d0449c >2012-12-17 08:46:07,806 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68c42198 >2012-12-17 08:46:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:07,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e56325c >2012-12-17 08:46:07,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e56325c >2012-12-17 08:46:07,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68c42198 >2012-12-17 08:46:07,826 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:13,981 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:46:16,679 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7f13ba47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:46:16,791 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1a0f716] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:46:16,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39ad0a19 >2012-12-17 08:46:16,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f001e6e >2012-12-17 08:46:16,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:17,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, SpmStopVDSCommand, log id: 6f001e6e >2012-12-17 08:46:17,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, ResetIrsVDSCommand, log id: 39ad0a19 >2012-12-17 08:46:17,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1642bcba >2012-12-17 08:46:17,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:17,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1642bcba >2012-12-17 08:46:24,173 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:46:27,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:27,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4645 >2012-12-17 08:46:27,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4645, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6eddfb7c >2012-12-17 08:46:27,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling started: taskId = 0d2dc974-8736-4330-82bb-fa9488b08200 >2012-12-17 08:46:28,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling ended: taskId = 0d2dc974-8736-4330-82bb-fa9488b08200 task status = finished >2012-12-17 08:46:28,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] spmStart polling ended, spm status: SPM >2012-12-17 08:46:28,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [27478f27] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0d2dc974-8736-4330-82bb-fa9488b08200), log id: 539b4a27 >2012-12-17 08:46:28,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, HSMClearTaskVDSCommand, log id: 539b4a27 >2012-12-17 08:46:28,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [27478f27] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78217152, log id: 6eddfb7c >2012-12-17 08:46:28,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:28,361 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:46:28,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f5846ee >2012-12-17 08:46:28,388 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [27478f27] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:46:28,490 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [d00852e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:46:28,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d6cf6db >2012-12-17 08:46:28,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11ec7931 >2012-12-17 08:46:28,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:28,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, SpmStopVDSCommand, log id: 11ec7931 >2012-12-17 08:46:28,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, ResetIrsVDSCommand, log id: 3d6cf6db >2012-12-17 08:46:28,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e8d1dec >2012-12-17 08:46:28,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:28,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e8d1dec >2012-12-17 08:46:29,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:29,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4646 >2012-12-17 08:46:29,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4646, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f7c759a >2012-12-17 08:46:29,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = 2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0 >2012-12-17 08:46:30,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = 2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0 task status = finished >2012-12-17 08:46:30,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM >2012-12-17 08:46:30,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2711d0a9-575f-4601-a3e0-2c7bf4f3c0f0), log id: 2a085f3c >2012-12-17 08:46:30,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 2a085f3c >2012-12-17 08:46:30,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@ba471b3, log id: 3f7c759a >2012-12-17 08:46:30,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:30,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:30,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 737d76b1 >2012-12-17 08:46:30,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 737d76b1 >2012-12-17 08:46:30,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f5846ee >2012-12-17 08:46:30,212 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:30,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d27d067 >2012-12-17 08:46:30,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:30,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7063efd7 >2012-12-17 08:46:30,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7063efd7 >2012-12-17 08:46:30,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d27d067 >2012-12-17 08:46:30,237 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:34,364 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:46:39,119 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5970076c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:46:39,423 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [75b7a65b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:46:39,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 676a499c >2012-12-17 08:46:39,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e0523b8 >2012-12-17 08:46:39,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:39,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStopVDSCommand, log id: 2e0523b8 >2012-12-17 08:46:39,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, ResetIrsVDSCommand, log id: 676a499c >2012-12-17 08:46:39,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ef39ce8 >2012-12-17 08:46:39,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:39,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ef39ce8 >2012-12-17 08:46:44,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:46:49,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:50,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4647 >2012-12-17 08:46:50,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4647, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62c94fb2 >2012-12-17 08:46:50,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling started: taskId = 74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691 >2012-12-17 08:46:51,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling ended: taskId = 74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691 task status = finished >2012-12-17 08:46:51,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] spmStart polling ended, spm status: SPM >2012-12-17 08:46:51,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=74bbd10d-2ace-4bd1-bd5e-b5f5b4ec8691), log id: 7816b8f1 >2012-12-17 08:46:51,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, HSMClearTaskVDSCommand, log id: 7816b8f1 >2012-12-17 08:46:51,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5b5bb621] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bc813cf, log id: 62c94fb2 >2012-12-17 08:46:51,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:51,148 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:46:51,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 764e3aef >2012-12-17 08:46:51,213 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b5bb621] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:46:51,385 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [71efd086] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:46:51,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69056b4 >2012-12-17 08:46:51,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4440f193 >2012-12-17 08:46:51,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:51,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, SpmStopVDSCommand, log id: 4440f193 >2012-12-17 08:46:51,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, ResetIrsVDSCommand, log id: 69056b4 >2012-12-17 08:46:51,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21dbfb83 >2012-12-17 08:46:51,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:51,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21dbfb83 >2012-12-17 08:46:51,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:46:51,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4648 >2012-12-17 08:46:51,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4648, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5737c9c >2012-12-17 08:46:51,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling started: taskId = e7ac015c-3dbe-49cb-a066-159879c3d1d5 >2012-12-17 08:46:52,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended: taskId = e7ac015c-3dbe-49cb-a066-159879c3d1d5 task status = finished >2012-12-17 08:46:52,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended, spm status: SPM >2012-12-17 08:46:52,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7ac015c-3dbe-49cb-a066-159879c3d1d5), log id: 56a8d7eb >2012-12-17 08:46:52,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMClearTaskVDSCommand, log id: 56a8d7eb >2012-12-17 08:46:52,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66f7b1bf, log id: 5737c9c >2012-12-17 08:46:52,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:46:52,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:52,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 373b2aac >2012-12-17 08:46:52,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 373b2aac >2012-12-17 08:46:52,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 764e3aef >2012-12-17 08:46:52,931 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:52,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6e9e88 >2012-12-17 08:46:52,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:46:52,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b8c9512 >2012-12-17 08:46:52,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b8c9512 >2012-12-17 08:46:52,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6e9e88 >2012-12-17 08:46:52,952 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [50f32e72] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:46:54,747 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:47:01,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [2649fcd5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:02,042 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:02,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5fc734e1 >2012-12-17 08:47:02,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64e79e64 >2012-12-17 08:47:02,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:02,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStopVDSCommand, log id: 64e79e64 >2012-12-17 08:47:02,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, ResetIrsVDSCommand, log id: 5fc734e1 >2012-12-17 08:47:02,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53a984a1 >2012-12-17 08:47:02,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:02,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53a984a1 >2012-12-17 08:47:04,917 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:47:12,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:12,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4649 >2012-12-17 08:47:12,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4649, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8f8fb4c >2012-12-17 08:47:12,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling started: taskId = 19c08f48-cb51-4489-b783-5fb3c6afc42f >2012-12-17 08:47:13,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended: taskId = 19c08f48-cb51-4489-b783-5fb3c6afc42f task status = finished >2012-12-17 08:47:13,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] spmStart polling ended, spm status: SPM >2012-12-17 08:47:13,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19c08f48-cb51-4489-b783-5fb3c6afc42f), log id: 5245350d >2012-12-17 08:47:13,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, HSMClearTaskVDSCommand, log id: 5245350d >2012-12-17 08:47:13,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [51b75138] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d9e12ad, log id: 8f8fb4c >2012-12-17 08:47:13,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:47:13,707 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:47:13,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b71578a >2012-12-17 08:47:13,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51b75138] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:14,039 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [726a9e10] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:14,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a78ee7 >2012-12-17 08:47:14,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d304603 >2012-12-17 08:47:14,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:14,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, SpmStopVDSCommand, log id: 4d304603 >2012-12-17 08:47:14,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, ResetIrsVDSCommand, log id: 17a78ee7 >2012-12-17 08:47:14,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40dcfb98 >2012-12-17 08:47:14,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:14,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [726a9e10] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40dcfb98 >2012-12-17 08:47:14,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:14,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4650 >2012-12-17 08:47:14,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4650, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 139a354b >2012-12-17 08:47:14,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling started: taskId = f7af5754-eb53-4cb3-aa4e-c72e6c252366 >2012-12-17 08:47:16,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling ended: taskId = f7af5754-eb53-4cb3-aa4e-c72e6c252366 task status = finished >2012-12-17 08:47:16,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] spmStart polling ended, spm status: SPM >2012-12-17 08:47:16,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7af5754-eb53-4cb3-aa4e-c72e6c252366), log id: 77d79ef0 >2012-12-17 08:47:16,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMClearTaskVDSCommand, log id: 77d79ef0 >2012-12-17 08:47:16,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d03dc07, log id: 139a354b >2012-12-17 08:47:16,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:47:16,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:16,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e07d210 >2012-12-17 08:47:16,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e07d210 >2012-12-17 08:47:16,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b71578a >2012-12-17 08:47:16,353 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:47:16,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73703840 >2012-12-17 08:47:16,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:16,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a592ce0 >2012-12-17 08:47:16,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a592ce0 >2012-12-17 08:47:16,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [783fb511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73703840 >2012-12-17 08:47:16,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [783fb511] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:47:24,508 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [566819d6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:24,620 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [388f9500] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:24,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 532c6f09 >2012-12-17 08:47:24,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79b09590 >2012-12-17 08:47:24,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:24,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, SpmStopVDSCommand, log id: 79b09590 >2012-12-17 08:47:24,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, ResetIrsVDSCommand, log id: 532c6f09 >2012-12-17 08:47:24,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4baa33a9 >2012-12-17 08:47:24,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:24,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4baa33a9 >2012-12-17 08:47:25,263 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:47:35,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:35,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4651 >2012-12-17 08:47:35,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4651, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bc77fe5 >2012-12-17 08:47:35,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling started: taskId = 7753b6d7-716e-4b34-912f-ff51b3790a76 >2012-12-17 08:47:36,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling ended: taskId = 7753b6d7-716e-4b34-912f-ff51b3790a76 task status = finished >2012-12-17 08:47:36,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] spmStart polling ended, spm status: SPM >2012-12-17 08:47:36,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7753b6d7-716e-4b34-912f-ff51b3790a76), log id: 4a8b2a07 >2012-12-17 08:47:36,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, HSMClearTaskVDSCommand, log id: 4a8b2a07 >2012-12-17 08:47:36,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [4595e4ad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@767457cc, log id: 2bc77fe5 >2012-12-17 08:47:36,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:47:36,177 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:47:36,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34332a3a >2012-12-17 08:47:36,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4595e4ad] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:36,300 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [2de700da] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:36,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71442728 >2012-12-17 08:47:36,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a53a09b >2012-12-17 08:47:36,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:36,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, SpmStopVDSCommand, log id: 6a53a09b >2012-12-17 08:47:36,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, ResetIrsVDSCommand, log id: 71442728 >2012-12-17 08:47:36,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2de700da] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e4fbec5 >2012-12-17 08:47:36,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2de700da] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:36,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2de700da] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e4fbec5 >2012-12-17 08:47:36,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:36,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4652 >2012-12-17 08:47:36,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4652, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78eb65fd >2012-12-17 08:47:36,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling started: taskId = a7273340-85a6-4978-b25f-4dab207a2637 >2012-12-17 08:47:37,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling ended: taskId = a7273340-85a6-4978-b25f-4dab207a2637 task status = finished >2012-12-17 08:47:37,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] spmStart polling ended, spm status: SPM >2012-12-17 08:47:37,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7273340-85a6-4978-b25f-4dab207a2637), log id: 22ab54c1 >2012-12-17 08:47:37,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMClearTaskVDSCommand, log id: 22ab54c1 >2012-12-17 08:47:37,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e495caf, log id: 78eb65fd >2012-12-17 08:47:37,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:47:37,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:37,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4155716d >2012-12-17 08:47:37,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4155716d >2012-12-17 08:47:37,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34332a3a >2012-12-17 08:47:37,797 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:47:37,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5089cc9d >2012-12-17 08:47:37,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:37,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d360188 >2012-12-17 08:47:37,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d360188 >2012-12-17 08:47:37,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [4c63d330] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5089cc9d >2012-12-17 08:47:37,817 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [4c63d330] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:47:46,719 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [717cca29] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:46,756 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:47:46,894 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1bff0c70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:46,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24681b6 >2012-12-17 08:47:46,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 444dddc1 >2012-12-17 08:47:46,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:47,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SpmStopVDSCommand, log id: 444dddc1 >2012-12-17 08:47:47,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, ResetIrsVDSCommand, log id: 24681b6 >2012-12-17 08:47:47,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39131659 >2012-12-17 08:47:47,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:47,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39131659 >2012-12-17 08:47:57,071 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:47:57,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:57,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4653 >2012-12-17 08:47:57,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4653, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7be4d3a >2012-12-17 08:47:57,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling started: taskId = 72f19c36-523d-4a5c-b296-989215321f55 >2012-12-17 08:47:58,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling ended: taskId = 72f19c36-523d-4a5c-b296-989215321f55 task status = finished >2012-12-17 08:47:58,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] spmStart polling ended, spm status: SPM >2012-12-17 08:47:58,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72f19c36-523d-4a5c-b296-989215321f55), log id: 3e05899a >2012-12-17 08:47:58,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, HSMClearTaskVDSCommand, log id: 3e05899a >2012-12-17 08:47:58,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [68b5cbc1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d8cfd45, log id: 7be4d3a >2012-12-17 08:47:58,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:47:58,464 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:47:58,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e9abb6e >2012-12-17 08:47:58,480 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [68b5cbc1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:47:58,603 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [446aaffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:47:58,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 86ccb70 >2012-12-17 08:47:58,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40b48f80 >2012-12-17 08:47:58,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:59,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStopVDSCommand, log id: 40b48f80 >2012-12-17 08:47:59,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, ResetIrsVDSCommand, log id: 86ccb70 >2012-12-17 08:47:59,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a09f11d >2012-12-17 08:47:59,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:47:59,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a09f11d >2012-12-17 08:47:59,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:47:59,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4654 >2012-12-17 08:47:59,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4654, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 401e1417 >2012-12-17 08:47:59,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling started: taskId = ca85646b-67f1-4f48-b074-94f4b96c1fa3 >2012-12-17 08:48:00,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling ended: taskId = ca85646b-67f1-4f48-b074-94f4b96c1fa3 task status = finished >2012-12-17 08:48:00,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] spmStart polling ended, spm status: SPM >2012-12-17 08:48:00,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca85646b-67f1-4f48-b074-94f4b96c1fa3), log id: 251af9e0 >2012-12-17 08:48:00,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMClearTaskVDSCommand, log id: 251af9e0 >2012-12-17 08:48:00,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e0af428, log id: 401e1417 >2012-12-17 08:48:01,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:48:01,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:01,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19169891 >2012-12-17 08:48:01,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19169891 >2012-12-17 08:48:01,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e9abb6e >2012-12-17 08:48:01,315 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:01,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a7a6475 >2012-12-17 08:48:01,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:01,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 15b44e60 >2012-12-17 08:48:01,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 15b44e60 >2012-12-17 08:48:01,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [40a5fc59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a7a6475 >2012-12-17 08:48:01,339 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [40a5fc59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:07,300 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:48:09,397 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [381e4566] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:48:09,574 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [15ef2937] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:48:09,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 901e857 >2012-12-17 08:48:09,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 215a8047 >2012-12-17 08:48:09,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:09,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SpmStopVDSCommand, log id: 215a8047 >2012-12-17 08:48:09,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, ResetIrsVDSCommand, log id: 901e857 >2012-12-17 08:48:09,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d53f90c >2012-12-17 08:48:09,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:09,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d53f90c >2012-12-17 08:48:17,550 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:48:19,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:48:19,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4655 >2012-12-17 08:48:19,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4655, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ff68367 >2012-12-17 08:48:19,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling started: taskId = 2e129f9d-cc40-415f-a718-55759e3ec462 >2012-12-17 08:48:20,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling ended: taskId = 2e129f9d-cc40-415f-a718-55759e3ec462 task status = finished >2012-12-17 08:48:20,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] spmStart polling ended, spm status: SPM >2012-12-17 08:48:20,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e129f9d-cc40-415f-a718-55759e3ec462), log id: 55ff7930 >2012-12-17 08:48:21,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, HSMClearTaskVDSCommand, log id: 55ff7930 >2012-12-17 08:48:21,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [58b8166b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@776d6a6b, log id: 3ff68367 >2012-12-17 08:48:21,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:48:21,392 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:48:21,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4b74b3f3 >2012-12-17 08:48:21,423 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [58b8166b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:48:21,540 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [3969aa82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:48:21,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bd05288 >2012-12-17 08:48:21,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 452c700 >2012-12-17 08:48:21,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:21,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, SpmStopVDSCommand, log id: 452c700 >2012-12-17 08:48:21,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, ResetIrsVDSCommand, log id: 5bd05288 >2012-12-17 08:48:21,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 330f18fe >2012-12-17 08:48:21,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:21,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 330f18fe >2012-12-17 08:48:21,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:48:21,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4656 >2012-12-17 08:48:21,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4656, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 117925bb >2012-12-17 08:48:21,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling started: taskId = 20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c >2012-12-17 08:48:22,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling ended: taskId = 20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c task status = finished >2012-12-17 08:48:22,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] spmStart polling ended, spm status: SPM >2012-12-17 08:48:22,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20fe5b72-2b7e-48f5-b8eb-4a3dff33a28c), log id: 202fcbe9 >2012-12-17 08:48:22,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMClearTaskVDSCommand, log id: 202fcbe9 >2012-12-17 08:48:22,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21f43e81, log id: 117925bb >2012-12-17 08:48:23,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:48:23,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:23,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 991be02 >2012-12-17 08:48:23,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 991be02 >2012-12-17 08:48:23,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4b74b3f3 >2012-12-17 08:48:23,050 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:23,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f517a36 >2012-12-17 08:48:23,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:23,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2cdaacf7 >2012-12-17 08:48:23,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2cdaacf7 >2012-12-17 08:48:23,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-31) [107aac36] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f517a36 >2012-12-17 08:48:23,070 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-31) [107aac36] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:27,786 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:48:31,959 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7ad32f3e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:48:32,090 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [155e23b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:48:32,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75b19215 >2012-12-17 08:48:32,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b6327d9 >2012-12-17 08:48:32,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:32,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStopVDSCommand, log id: 7b6327d9 >2012-12-17 08:48:32,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, ResetIrsVDSCommand, log id: 75b19215 >2012-12-17 08:48:32,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 621b05be >2012-12-17 08:48:32,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:32,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 621b05be >2012-12-17 08:48:37,978 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:48:42,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:48:42,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4657 >2012-12-17 08:48:42,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4657, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ac81200 >2012-12-17 08:48:42,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling started: taskId = e0d4b294-cd04-4205-8f00-49241b117e8c >2012-12-17 08:48:43,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling ended: taskId = e0d4b294-cd04-4205-8f00-49241b117e8c task status = finished >2012-12-17 08:48:43,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] spmStart polling ended, spm status: SPM >2012-12-17 08:48:43,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e0d4b294-cd04-4205-8f00-49241b117e8c), log id: 33cbaeef >2012-12-17 08:48:43,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, HSMClearTaskVDSCommand, log id: 33cbaeef >2012-12-17 08:48:43,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [6f019f85] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@149e77a, log id: 1ac81200 >2012-12-17 08:48:43,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:48:43,634 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:48:43,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fe30d05 >2012-12-17 08:48:43,665 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [6f019f85] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:48:43,782 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [27a8ed97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:48:43,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 406ddfcc >2012-12-17 08:48:43,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34e0b9b >2012-12-17 08:48:43,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:44,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStopVDSCommand, log id: 34e0b9b >2012-12-17 08:48:44,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, ResetIrsVDSCommand, log id: 406ddfcc >2012-12-17 08:48:44,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 279c62a1 >2012-12-17 08:48:44,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:44,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 279c62a1 >2012-12-17 08:48:44,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:48:44,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4658 >2012-12-17 08:48:44,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4658, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 234667e2 >2012-12-17 08:48:44,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling started: taskId = 61d5bb19-cf76-41ff-b7e0-4d7988b9af70 >2012-12-17 08:48:45,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling ended: taskId = 61d5bb19-cf76-41ff-b7e0-4d7988b9af70 task status = finished >2012-12-17 08:48:45,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] spmStart polling ended, spm status: SPM >2012-12-17 08:48:45,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=61d5bb19-cf76-41ff-b7e0-4d7988b9af70), log id: 38de7a4b >2012-12-17 08:48:45,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMClearTaskVDSCommand, log id: 38de7a4b >2012-12-17 08:48:45,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@162eef18, log id: 234667e2 >2012-12-17 08:48:45,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:48:45,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:45,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d784584 >2012-12-17 08:48:45,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d784584 >2012-12-17 08:48:45,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fe30d05 >2012-12-17 08:48:45,267 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:45,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e31b49a >2012-12-17 08:48:45,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:45,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12dc2726 >2012-12-17 08:48:45,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12dc2726 >2012-12-17 08:48:45,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [786fe4b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e31b49a >2012-12-17 08:48:45,287 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [786fe4b5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:48:48,187 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:48:54,141 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [1ee6f70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:48:54,309 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [337c1daa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:48:54,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62e977b >2012-12-17 08:48:54,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a19bfb >2012-12-17 08:48:54,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:54,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, SpmStopVDSCommand, log id: 3a19bfb >2012-12-17 08:48:54,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, ResetIrsVDSCommand, log id: 62e977b >2012-12-17 08:48:54,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19a23671 >2012-12-17 08:48:54,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:48:54,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19a23671 >2012-12-17 08:48:58,379 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:49:04,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:04,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4659 >2012-12-17 08:49:04,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4659, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f4c9760 >2012-12-17 08:49:04,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling started: taskId = 9c644c2d-5181-4c61-8326-67a3a343bd75 >2012-12-17 08:49:05,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended: taskId = 9c644c2d-5181-4c61-8326-67a3a343bd75 task status = finished >2012-12-17 08:49:05,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] spmStart polling ended, spm status: SPM >2012-12-17 08:49:05,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c644c2d-5181-4c61-8326-67a3a343bd75), log id: 10a2e041 >2012-12-17 08:49:05,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, HSMClearTaskVDSCommand, log id: 10a2e041 >2012-12-17 08:49:05,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [2e21b599] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3530d3ef, log id: 7f4c9760 >2012-12-17 08:49:05,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:06,031 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:49:06,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5046331 >2012-12-17 08:49:06,065 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [2e21b599] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:49:06,168 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [1a40d676] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:49:06,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18a25764 >2012-12-17 08:49:06,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7efc42f4 >2012-12-17 08:49:06,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:06,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, SpmStopVDSCommand, log id: 7efc42f4 >2012-12-17 08:49:06,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, ResetIrsVDSCommand, log id: 18a25764 >2012-12-17 08:49:06,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30332130 >2012-12-17 08:49:06,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [1a40d676] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:06,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [1a40d676] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30332130 >2012-12-17 08:49:06,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:06,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4660 >2012-12-17 08:49:06,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4660, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 655dead0 >2012-12-17 08:49:06,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling started: taskId = 33254414-51e3-4227-b34e-675683f2940a >2012-12-17 08:49:07,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling ended: taskId = 33254414-51e3-4227-b34e-675683f2940a task status = finished >2012-12-17 08:49:07,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] spmStart polling ended, spm status: SPM >2012-12-17 08:49:07,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=33254414-51e3-4227-b34e-675683f2940a), log id: 566949d >2012-12-17 08:49:07,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMClearTaskVDSCommand, log id: 566949d >2012-12-17 08:49:07,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11a6d230, log id: 655dead0 >2012-12-17 08:49:07,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:07,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:07,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2ad7bd58 >2012-12-17 08:49:07,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ad7bd58 >2012-12-17 08:49:07,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5046331 >2012-12-17 08:49:07,701 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:07,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 331dbee >2012-12-17 08:49:07,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:07,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 248391d8 >2012-12-17 08:49:07,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 248391d8 >2012-12-17 08:49:07,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-6) [6cc0b445] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 331dbee >2012-12-17 08:49:07,720 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-6) [6cc0b445] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:08,617 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:49:16,539 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [28e17066] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:49:16,711 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [108329de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:49:16,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 690dfb4 >2012-12-17 08:49:16,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f561792 >2012-12-17 08:49:16,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:16,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, SpmStopVDSCommand, log id: 6f561792 >2012-12-17 08:49:16,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, ResetIrsVDSCommand, log id: 690dfb4 >2012-12-17 08:49:16,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78504d03 >2012-12-17 08:49:16,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:16,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78504d03 >2012-12-17 08:49:18,925 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:49:27,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:27,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4661 >2012-12-17 08:49:27,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4661, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c2a2fd2 >2012-12-17 08:49:27,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling started: taskId = 3320270d-0705-4814-bcc9-5a8ee20a73cc >2012-12-17 08:49:28,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling ended: taskId = 3320270d-0705-4814-bcc9-5a8ee20a73cc task status = finished >2012-12-17 08:49:28,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] spmStart polling ended, spm status: SPM >2012-12-17 08:49:28,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3320270d-0705-4814-bcc9-5a8ee20a73cc), log id: 5879adb7 >2012-12-17 08:49:28,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, HSMClearTaskVDSCommand, log id: 5879adb7 >2012-12-17 08:49:28,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5f1742db] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c7bf6c2, log id: 4c2a2fd2 >2012-12-17 08:49:28,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:28,325 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:49:28,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 583b9aa8 >2012-12-17 08:49:28,359 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5f1742db] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:49:28,560 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5bc809fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:49:28,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 958fa36 >2012-12-17 08:49:28,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 59e56d11 >2012-12-17 08:49:28,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:28,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStopVDSCommand, log id: 59e56d11 >2012-12-17 08:49:28,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, ResetIrsVDSCommand, log id: 958fa36 >2012-12-17 08:49:28,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 449a9d13 >2012-12-17 08:49:28,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:28,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 449a9d13 >2012-12-17 08:49:29,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:29,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4662 >2012-12-17 08:49:29,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4662, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: aa10928 >2012-12-17 08:49:29,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling started: taskId = 33c5dd4d-7903-4f82-b753-91e4a7499ea8 >2012-12-17 08:49:30,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended: taskId = 33c5dd4d-7903-4f82-b753-91e4a7499ea8 task status = finished >2012-12-17 08:49:30,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] spmStart polling ended, spm status: SPM >2012-12-17 08:49:30,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=33c5dd4d-7903-4f82-b753-91e4a7499ea8), log id: b282da3 >2012-12-17 08:49:30,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMClearTaskVDSCommand, log id: b282da3 >2012-12-17 08:49:30,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5683d847, log id: aa10928 >2012-12-17 08:49:30,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:30,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:30,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7008a527 >2012-12-17 08:49:30,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7008a527 >2012-12-17 08:49:30,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 583b9aa8 >2012-12-17 08:49:30,261 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:30,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4792dce7 >2012-12-17 08:49:30,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:30,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bef4002 >2012-12-17 08:49:30,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bef4002 >2012-12-17 08:49:30,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [1f4324c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4792dce7 >2012-12-17 08:49:30,281 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [1f4324c3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:39,074 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [bb53630] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:49:39,180 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:49:39,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ae6fbe >2012-12-17 08:49:39,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 56324eb2 >2012-12-17 08:49:39,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:39,354 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:49:39,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, SpmStopVDSCommand, log id: 56324eb2 >2012-12-17 08:49:39,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, ResetIrsVDSCommand, log id: 13ae6fbe >2012-12-17 08:49:39,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41874892 >2012-12-17 08:49:39,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:39,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41874892 >2012-12-17 08:49:49,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:49,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4663 >2012-12-17 08:49:49,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4663, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c519af6 >2012-12-17 08:49:49,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling started: taskId = 3174d548-40a3-40b7-9020-a87b5479b4b9 >2012-12-17 08:49:50,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended: taskId = 3174d548-40a3-40b7-9020-a87b5479b4b9 task status = finished >2012-12-17 08:49:50,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] spmStart polling ended, spm status: SPM >2012-12-17 08:49:50,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3174d548-40a3-40b7-9020-a87b5479b4b9), log id: 2fafe119 >2012-12-17 08:49:50,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, HSMClearTaskVDSCommand, log id: 2fafe119 >2012-12-17 08:49:50,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [6e9d510a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43e348c4, log id: 6c519af6 >2012-12-17 08:49:50,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:51,173 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:49:51,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 736131e3 >2012-12-17 08:49:51,194 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [6e9d510a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:49:51,331 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [63fee526] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:49:51,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8677827 >2012-12-17 08:49:51,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 43e829f7 >2012-12-17 08:49:51,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:51,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStopVDSCommand, log id: 43e829f7 >2012-12-17 08:49:51,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, ResetIrsVDSCommand, log id: 8677827 >2012-12-17 08:49:51,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6fe84348 >2012-12-17 08:49:51,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:51,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6fe84348 >2012-12-17 08:49:51,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:49:51,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4664 >2012-12-17 08:49:51,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4664, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45b5af50 >2012-12-17 08:49:51,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 47da8b06-04e0-4744-b6dc-1625b9fddd5f >2012-12-17 08:49:52,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 47da8b06-04e0-4744-b6dc-1625b9fddd5f task status = finished >2012-12-17 08:49:52,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM >2012-12-17 08:49:52,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=47da8b06-04e0-4744-b6dc-1625b9fddd5f), log id: 1725fa2d >2012-12-17 08:49:52,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 1725fa2d >2012-12-17 08:49:52,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e4bac3a, log id: 45b5af50 >2012-12-17 08:49:52,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:49:52,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:52,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ddfa345 >2012-12-17 08:49:52,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ddfa345 >2012-12-17 08:49:52,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 736131e3 >2012-12-17 08:49:52,851 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:52,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 45581655 >2012-12-17 08:49:52,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:49:52,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fa22197 >2012-12-17 08:49:52,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fa22197 >2012-12-17 08:49:52,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 45581655 >2012-12-17 08:49:52,872 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:49:59,810 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Checking autorecoverable hosts >2012-12-17 08:50:00,061 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Autorecovering 0 hosts >2012-12-17 08:50:00,061 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Checking autorecoverable hosts done >2012-12-17 08:50:00,061 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [34599a45] Autorecovering storage domains is disabled, skipping >2012-12-17 08:50:02,246 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [6acb626c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:50:02,795 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [1756fb88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:50:02,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 290e135d >2012-12-17 08:50:02,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38a799b9 >2012-12-17 08:50:02,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:03,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStopVDSCommand, log id: 38a799b9 >2012-12-17 08:50:03,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, ResetIrsVDSCommand, log id: 290e135d >2012-12-17 08:50:03,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41c82391 >2012-12-17 08:50:03,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:03,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41c82391 >2012-12-17 08:50:10,009 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:13,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:50:13,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4665 >2012-12-17 08:50:13,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4665, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f0b70ca >2012-12-17 08:50:13,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling started: taskId = 7f5cfcc3-b93f-451e-bd95-3af4d917abef >2012-12-17 08:50:14,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling ended: taskId = 7f5cfcc3-b93f-451e-bd95-3af4d917abef task status = finished >2012-12-17 08:50:14,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] spmStart polling ended, spm status: SPM >2012-12-17 08:50:14,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [30867547] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7f5cfcc3-b93f-451e-bd95-3af4d917abef), log id: 419bf76 >2012-12-17 08:50:14,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, HSMClearTaskVDSCommand, log id: 419bf76 >2012-12-17 08:50:14,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [30867547] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3668de14, log id: 4f0b70ca >2012-12-17 08:50:14,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:50:14,269 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:50:14,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 70a94de7 >2012-12-17 08:50:14,280 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [30867547] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:50:14,377 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [69834199] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:50:14,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 59c3e568 >2012-12-17 08:50:14,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 467d720b >2012-12-17 08:50:14,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:14,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStopVDSCommand, log id: 467d720b >2012-12-17 08:50:14,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, ResetIrsVDSCommand, log id: 59c3e568 >2012-12-17 08:50:14,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1614edb >2012-12-17 08:50:14,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:14,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1614edb >2012-12-17 08:50:14,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:50:14,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4666 >2012-12-17 08:50:14,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4666, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 548f926b >2012-12-17 08:50:14,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = 201dfa98-b024-412f-ae6a-8fee6a95dde1 >2012-12-17 08:50:15,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = 201dfa98-b024-412f-ae6a-8fee6a95dde1 task status = finished >2012-12-17 08:50:15,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM >2012-12-17 08:50:15,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=201dfa98-b024-412f-ae6a-8fee6a95dde1), log id: 3dd06ab8 >2012-12-17 08:50:15,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 3dd06ab8 >2012-12-17 08:50:15,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e0e75bc, log id: 548f926b >2012-12-17 08:50:15,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:50:16,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:16,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 651cf3d2 >2012-12-17 08:50:16,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 651cf3d2 >2012-12-17 08:50:16,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 70a94de7 >2012-12-17 08:50:16,098 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:50:16,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7c10555a >2012-12-17 08:50:16,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:16,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48b789cd >2012-12-17 08:50:16,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48b789cd >2012-12-17 08:50:16,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7c10555a >2012-12-17 08:50:16,118 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [28319bba] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:50:20,238 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:24,778 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [40a5fc59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:50:24,935 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [5abe3acd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:50:25,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 670b31a5 >2012-12-17 08:50:25,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5573426a >2012-12-17 08:50:25,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:25,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, SpmStopVDSCommand, log id: 5573426a >2012-12-17 08:50:26,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, ResetIrsVDSCommand, log id: 670b31a5 >2012-12-17 08:50:26,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40a778e4 >2012-12-17 08:50:26,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:26,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40a778e4 >2012-12-17 08:50:31,158 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:36,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:50:36,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4667 >2012-12-17 08:50:36,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4667, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 209d7ea0 >2012-12-17 08:50:36,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling started: taskId = e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42 >2012-12-17 08:50:37,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling ended: taskId = e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42 task status = finished >2012-12-17 08:50:37,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] spmStart polling ended, spm status: SPM >2012-12-17 08:50:37,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e5da671c-aacf-4d8c-a8a7-7bc06a4d7e42), log id: 790b80b0 >2012-12-17 08:50:37,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, HSMClearTaskVDSCommand, log id: 790b80b0 >2012-12-17 08:50:37,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [5b666f42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@622de0e5, log id: 209d7ea0 >2012-12-17 08:50:37,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:50:37,819 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:50:37,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7498f57b >2012-12-17 08:50:37,835 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [5b666f42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:50:37,965 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [3ef697d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:50:38,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9902847 >2012-12-17 08:50:38,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 72adf961 >2012-12-17 08:50:38,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:38,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, SpmStopVDSCommand, log id: 72adf961 >2012-12-17 08:50:38,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, ResetIrsVDSCommand, log id: 9902847 >2012-12-17 08:50:38,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c2fb07f >2012-12-17 08:50:38,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:38,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c2fb07f >2012-12-17 08:50:38,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:50:38,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4668 >2012-12-17 08:50:38,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4668, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7722fc7 >2012-12-17 08:50:38,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling started: taskId = 41b3e09c-c53f-448d-baaf-d51e676b3ec1 >2012-12-17 08:50:39,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended: taskId = 41b3e09c-c53f-448d-baaf-d51e676b3ec1 task status = finished >2012-12-17 08:50:39,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended, spm status: SPM >2012-12-17 08:50:39,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=41b3e09c-c53f-448d-baaf-d51e676b3ec1), log id: 559511e2 >2012-12-17 08:50:39,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMClearTaskVDSCommand, log id: 559511e2 >2012-12-17 08:50:39,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2dc93853, log id: 7722fc7 >2012-12-17 08:50:39,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:50:39,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:39,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5f2902b6 >2012-12-17 08:50:39,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5f2902b6 >2012-12-17 08:50:39,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7498f57b >2012-12-17 08:50:39,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:50:39,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b2e281f >2012-12-17 08:50:39,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:39,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72eab901 >2012-12-17 08:50:39,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72eab901 >2012-12-17 08:50:39,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b2e281f >2012-12-17 08:50:39,469 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) [e4c8c37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:50:41,345 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:48,341 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [4c63d330] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:50:48,443 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [5dcd105e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:50:48,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3fef855f >2012-12-17 08:50:48,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7cece9e6 >2012-12-17 08:50:48,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:48,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SpmStopVDSCommand, log id: 7cece9e6 >2012-12-17 08:50:48,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, ResetIrsVDSCommand, log id: 3fef855f >2012-12-17 08:50:48,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bafbf3d >2012-12-17 08:50:48,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:50:48,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bafbf3d >2012-12-17 08:50:51,875 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:50:58,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:50:58,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4669 >2012-12-17 08:50:58,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4669, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 467b21e3 >2012-12-17 08:50:58,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling started: taskId = ca01353f-a28b-4611-82be-a3becdf565f2 >2012-12-17 08:50:59,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended: taskId = ca01353f-a28b-4611-82be-a3becdf565f2 task status = finished >2012-12-17 08:50:59,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] spmStart polling ended, spm status: SPM >2012-12-17 08:50:59,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca01353f-a28b-4611-82be-a3becdf565f2), log id: 1b040dbd >2012-12-17 08:51:00,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, HSMClearTaskVDSCommand, log id: 1b040dbd >2012-12-17 08:51:00,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [43f46040] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ad60580, log id: 467b21e3 >2012-12-17 08:51:00,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:00,046 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:51:00,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5488f4 >2012-12-17 08:51:00,073 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [43f46040] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:01,034 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [dd34b5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:01,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 598aee86 >2012-12-17 08:51:01,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1667a46d >2012-12-17 08:51:01,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:01,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, SpmStopVDSCommand, log id: 1667a46d >2012-12-17 08:51:01,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, ResetIrsVDSCommand, log id: 598aee86 >2012-12-17 08:51:01,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 421a6141 >2012-12-17 08:51:01,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [dd34b5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:01,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [dd34b5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 421a6141 >2012-12-17 08:51:01,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:51:01,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4670 >2012-12-17 08:51:01,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4670, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6890411b >2012-12-17 08:51:01,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling started: taskId = f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea >2012-12-17 08:51:02,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended: taskId = f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea task status = finished >2012-12-17 08:51:02,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended, spm status: SPM >2012-12-17 08:51:02,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7fb0fba-a7f1-4bf8-a4a2-dbae638279ea), log id: 6dc764c4 >2012-12-17 08:51:02,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMClearTaskVDSCommand, log id: 6dc764c4 >2012-12-17 08:51:02,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3da8b82a, log id: 6890411b >2012-12-17 08:51:02,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:02,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:02,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d4ae3ec >2012-12-17 08:51:02,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d4ae3ec >2012-12-17 08:51:02,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5488f4 >2012-12-17 08:51:02,667 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:02,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2fb9a1c5 >2012-12-17 08:51:02,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:02,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3dd94d91 >2012-12-17 08:51:02,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3dd94d91 >2012-12-17 08:51:02,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2fb9a1c5 >2012-12-17 08:51:02,687 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [3e0f12eb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:11,548 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [726a9e10] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:11,693 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [45590d60] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:11,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7312cb8b >2012-12-17 08:51:11,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a77a8af >2012-12-17 08:51:11,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:11,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, SpmStopVDSCommand, log id: 7a77a8af >2012-12-17 08:51:11,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, ResetIrsVDSCommand, log id: 7312cb8b >2012-12-17 08:51:11,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7859e73e >2012-12-17 08:51:11,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:11,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7859e73e >2012-12-17 08:51:12,338 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:51:22,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:51:22,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4671 >2012-12-17 08:51:22,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4671, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 298af872 >2012-12-17 08:51:22,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling started: taskId = cae355e4-a638-4ff7-add6-41a6c91bea68 >2012-12-17 08:51:23,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling ended: taskId = cae355e4-a638-4ff7-add6-41a6c91bea68 task status = finished >2012-12-17 08:51:23,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] spmStart polling ended, spm status: SPM >2012-12-17 08:51:23,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cae355e4-a638-4ff7-add6-41a6c91bea68), log id: 53d4beca >2012-12-17 08:51:23,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, HSMClearTaskVDSCommand, log id: 53d4beca >2012-12-17 08:51:23,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3eec3b83] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3439eac5, log id: 298af872 >2012-12-17 08:51:23,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:23,318 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:51:23,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cb88bf1 >2012-12-17 08:51:23,331 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3eec3b83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:23,467 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [7c1d0790] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:23,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2b94a9eb >2012-12-17 08:51:23,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5de3b838 >2012-12-17 08:51:23,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:24,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, SpmStopVDSCommand, log id: 5de3b838 >2012-12-17 08:51:24,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, ResetIrsVDSCommand, log id: 2b94a9eb >2012-12-17 08:51:24,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54fc8a0b >2012-12-17 08:51:24,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [7c1d0790] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:24,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [7c1d0790] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54fc8a0b >2012-12-17 08:51:24,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:51:24,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4672 >2012-12-17 08:51:24,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4672, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52c928a6 >2012-12-17 08:51:24,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = 8ee30c57-2f44-4a0a-953e-1896ad25efb7 >2012-12-17 08:51:25,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = 8ee30c57-2f44-4a0a-953e-1896ad25efb7 task status = finished >2012-12-17 08:51:25,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM >2012-12-17 08:51:25,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ee30c57-2f44-4a0a-953e-1896ad25efb7), log id: 10e93c40 >2012-12-17 08:51:25,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: 10e93c40 >2012-12-17 08:51:25,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4072966c, log id: 52c928a6 >2012-12-17 08:51:25,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:26,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:26,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1393007d >2012-12-17 08:51:26,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1393007d >2012-12-17 08:51:26,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cb88bf1 >2012-12-17 08:51:26,180 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:26,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 18051895 >2012-12-17 08:51:26,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:26,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7faa7d13 >2012-12-17 08:51:26,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7faa7d13 >2012-12-17 08:51:26,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 18051895 >2012-12-17 08:51:26,199 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:32,836 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:51:34,582 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2de700da] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:34,792 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [6d961413] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:34,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3eb436e3 >2012-12-17 08:51:34,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e94f6f9 >2012-12-17 08:51:34,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:35,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SpmStopVDSCommand, log id: 1e94f6f9 >2012-12-17 08:51:35,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, ResetIrsVDSCommand, log id: 3eb436e3 >2012-12-17 08:51:35,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b17ab52 >2012-12-17 08:51:35,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:35,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b17ab52 >2012-12-17 08:51:43,015 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:51:45,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:51:45,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4673 >2012-12-17 08:51:45,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4673, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72ddc50 >2012-12-17 08:51:45,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling started: taskId = aeded897-fd9b-46e7-9fbf-49ee29f65840 >2012-12-17 08:51:46,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling ended: taskId = aeded897-fd9b-46e7-9fbf-49ee29f65840 task status = finished >2012-12-17 08:51:46,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] spmStart polling ended, spm status: SPM >2012-12-17 08:51:46,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aeded897-fd9b-46e7-9fbf-49ee29f65840), log id: 6c72c6ff >2012-12-17 08:51:46,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, HSMClearTaskVDSCommand, log id: 6c72c6ff >2012-12-17 08:51:46,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [369f35cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6366c6f8, log id: 72ddc50 >2012-12-17 08:51:46,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:46,312 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:51:46,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68588ca1 >2012-12-17 08:51:46,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [369f35cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:46,584 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:46,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1219563a >2012-12-17 08:51:46,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 188b444c >2012-12-17 08:51:46,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:47,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, SpmStopVDSCommand, log id: 188b444c >2012-12-17 08:51:47,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, ResetIrsVDSCommand, log id: 1219563a >2012-12-17 08:51:47,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2983705 >2012-12-17 08:51:47,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:47,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2983705 >2012-12-17 08:51:47,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:51:47,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4674 >2012-12-17 08:51:47,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4674, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a58acfc >2012-12-17 08:51:47,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling started: taskId = 4a1f375f-a968-4edd-9975-a7fae867e08a >2012-12-17 08:51:48,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling ended: taskId = 4a1f375f-a968-4edd-9975-a7fae867e08a task status = finished >2012-12-17 08:51:48,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] spmStart polling ended, spm status: SPM >2012-12-17 08:51:48,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a1f375f-a968-4edd-9975-a7fae867e08a), log id: 68291869 >2012-12-17 08:51:48,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMClearTaskVDSCommand, log id: 68291869 >2012-12-17 08:51:48,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6201caeb, log id: 2a58acfc >2012-12-17 08:51:48,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:51:48,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:48,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f776570 >2012-12-17 08:51:48,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f776570 >2012-12-17 08:51:48,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68588ca1 >2012-12-17 08:51:48,405 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:48,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 56ab5a0f >2012-12-17 08:51:48,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:48,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73e34196 >2012-12-17 08:51:48,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73e34196 >2012-12-17 08:51:48,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1bff0c70] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 56ab5a0f >2012-12-17 08:51:48,426 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1bff0c70] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:51:53,254 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:51:57,263 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [2b22186] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:51:57,409 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [4793062d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:51:57,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 720fb799 >2012-12-17 08:51:57,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 212f4563 >2012-12-17 08:51:57,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:57,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, SpmStopVDSCommand, log id: 212f4563 >2012-12-17 08:51:57,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, ResetIrsVDSCommand, log id: 720fb799 >2012-12-17 08:51:57,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6dfdefc6 >2012-12-17 08:51:57,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:51:57,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6dfdefc6 >2012-12-17 08:52:03,473 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:52:07,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:07,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4675 >2012-12-17 08:52:07,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4675, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: edd6759 >2012-12-17 08:52:07,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling started: taskId = 14cd4bd5-bd6a-4e22-81f3-da19916dcbd7 >2012-12-17 08:52:08,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended: taskId = 14cd4bd5-bd6a-4e22-81f3-da19916dcbd7 task status = finished >2012-12-17 08:52:08,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] spmStart polling ended, spm status: SPM >2012-12-17 08:52:08,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14cd4bd5-bd6a-4e22-81f3-da19916dcbd7), log id: 461e5ede >2012-12-17 08:52:08,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, HSMClearTaskVDSCommand, log id: 461e5ede >2012-12-17 08:52:08,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [169ff372] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73e664c8, log id: edd6759 >2012-12-17 08:52:08,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:09,021 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:52:09,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a08efa5 >2012-12-17 08:52:09,051 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [169ff372] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:52:09,178 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:52:09,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 362177e3 >2012-12-17 08:52:09,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79692fa8 >2012-12-17 08:52:09,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:09,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStopVDSCommand, log id: 79692fa8 >2012-12-17 08:52:09,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, ResetIrsVDSCommand, log id: 362177e3 >2012-12-17 08:52:09,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a047226 >2012-12-17 08:52:09,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:09,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a047226 >2012-12-17 08:52:09,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:09,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4676 >2012-12-17 08:52:09,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4676, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 347f8d5e >2012-12-17 08:52:09,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling started: taskId = 59f89343-8723-4562-904c-f5ad0dedd1be >2012-12-17 08:52:10,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling ended: taskId = 59f89343-8723-4562-904c-f5ad0dedd1be task status = finished >2012-12-17 08:52:10,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] spmStart polling ended, spm status: SPM >2012-12-17 08:52:10,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=59f89343-8723-4562-904c-f5ad0dedd1be), log id: 14cf435 >2012-12-17 08:52:10,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMClearTaskVDSCommand, log id: 14cf435 >2012-12-17 08:52:10,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6334c43d, log id: 347f8d5e >2012-12-17 08:52:11,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:11,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:11,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 641db1bc >2012-12-17 08:52:11,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 641db1bc >2012-12-17 08:52:11,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a08efa5 >2012-12-17 08:52:11,998 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:52:11,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 597fe0fd >2012-12-17 08:52:11,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:12,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5705fbd7 >2012-12-17 08:52:12,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5705fbd7 >2012-12-17 08:52:12,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [15ef2937] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 597fe0fd >2012-12-17 08:52:12,018 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [15ef2937] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:52:13,750 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:52:19,516 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [37434c11] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:52:19,674 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [61136831] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:52:19,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18969ea4 >2012-12-17 08:52:19,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25fddbea >2012-12-17 08:52:19,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:19,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStopVDSCommand, log id: 25fddbea >2012-12-17 08:52:19,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, ResetIrsVDSCommand, log id: 18969ea4 >2012-12-17 08:52:19,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ee7b32c >2012-12-17 08:52:19,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:19,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ee7b32c >2012-12-17 08:52:23,950 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:52:30,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:30,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4677 >2012-12-17 08:52:30,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4677, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10149c93 >2012-12-17 08:52:30,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling started: taskId = 9cf9966d-4f49-467f-b941-49e6cba3d5bb >2012-12-17 08:52:31,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended: taskId = 9cf9966d-4f49-467f-b941-49e6cba3d5bb task status = finished >2012-12-17 08:52:31,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] spmStart polling ended, spm status: SPM >2012-12-17 08:52:31,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9cf9966d-4f49-467f-b941-49e6cba3d5bb), log id: 79d3af9d >2012-12-17 08:52:31,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, HSMClearTaskVDSCommand, log id: 79d3af9d >2012-12-17 08:52:31,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [1143d064] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38aeaae7, log id: 10149c93 >2012-12-17 08:52:31,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:31,569 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:52:31,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31b0b7a0 >2012-12-17 08:52:31,661 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1143d064] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:52:31,924 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:52:32,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7bccd9f9 >2012-12-17 08:52:32,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1c99c56b >2012-12-17 08:52:32,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:32,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStopVDSCommand, log id: 1c99c56b >2012-12-17 08:52:32,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, ResetIrsVDSCommand, log id: 7bccd9f9 >2012-12-17 08:52:32,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ff1f3be >2012-12-17 08:52:32,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:32,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ff1f3be >2012-12-17 08:52:32,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:32,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4678 >2012-12-17 08:52:32,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4678, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a132d9 >2012-12-17 08:52:32,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling started: taskId = 6950ea93-afd3-49c9-bff1-833f6ee2b905 >2012-12-17 08:52:33,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended: taskId = 6950ea93-afd3-49c9-bff1-833f6ee2b905 task status = finished >2012-12-17 08:52:33,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended, spm status: SPM >2012-12-17 08:52:33,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6950ea93-afd3-49c9-bff1-833f6ee2b905), log id: be3048a >2012-12-17 08:52:33,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMClearTaskVDSCommand, log id: be3048a >2012-12-17 08:52:33,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@653f061e, log id: 3a132d9 >2012-12-17 08:52:33,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:33,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:33,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 36f7bdd1 >2012-12-17 08:52:33,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 36f7bdd1 >2012-12-17 08:52:33,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31b0b7a0 >2012-12-17 08:52:33,601 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:52:33,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 242c02fc >2012-12-17 08:52:33,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:33,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 66c07d6a >2012-12-17 08:52:33,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66c07d6a >2012-12-17 08:52:33,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 242c02fc >2012-12-17 08:52:33,621 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [155e23b1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:52:34,158 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:52:42,467 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1f4324c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:52:42,567 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [164629b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:52:42,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 515e0f1e >2012-12-17 08:52:42,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ee04a3a >2012-12-17 08:52:42,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:43,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, SpmStopVDSCommand, log id: 5ee04a3a >2012-12-17 08:52:43,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, ResetIrsVDSCommand, log id: 515e0f1e >2012-12-17 08:52:43,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48b972e6 >2012-12-17 08:52:43,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:43,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48b972e6 >2012-12-17 08:52:44,319 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:52:53,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:53,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4679 >2012-12-17 08:52:53,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4679, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31fde7a6 >2012-12-17 08:52:53,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling started: taskId = 4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a >2012-12-17 08:52:54,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling ended: taskId = 4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a task status = finished >2012-12-17 08:52:54,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] spmStart polling ended, spm status: SPM >2012-12-17 08:52:54,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4cfd75cb-5dfb-46f9-bdce-e657fbf19c8a), log id: 7fc9c942 >2012-12-17 08:52:54,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, HSMClearTaskVDSCommand, log id: 7fc9c942 >2012-12-17 08:52:54,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [369a76fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18bc4d99, log id: 31fde7a6 >2012-12-17 08:52:54,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:54,511 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:52:54,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c7e99ef >2012-12-17 08:52:54,553 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [369a76fa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:52:54,768 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [214f1ee4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:52:54,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4b77eddd >2012-12-17 08:52:54,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65f44d4d >2012-12-17 08:52:54,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:54,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStopVDSCommand, log id: 65f44d4d >2012-12-17 08:52:55,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, ResetIrsVDSCommand, log id: 4b77eddd >2012-12-17 08:52:55,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 542352da >2012-12-17 08:52:55,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:55,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 542352da >2012-12-17 08:52:55,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:52:55,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4680 >2012-12-17 08:52:55,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4680, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b932c37 >2012-12-17 08:52:55,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling started: taskId = 8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1 >2012-12-17 08:52:56,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended: taskId = 8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1 task status = finished >2012-12-17 08:52:56,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended, spm status: SPM >2012-12-17 08:52:56,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8bf086ef-dddf-4b28-b4c8-b9fea5ee01f1), log id: 106556c >2012-12-17 08:52:56,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMClearTaskVDSCommand, log id: 106556c >2012-12-17 08:52:56,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46d225fa, log id: 6b932c37 >2012-12-17 08:52:56,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:52:56,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:56,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 548138fd >2012-12-17 08:52:56,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 548138fd >2012-12-17 08:52:56,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c7e99ef >2012-12-17 08:52:56,466 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:52:56,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fc49562 >2012-12-17 08:52:56,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:52:56,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 98f257e >2012-12-17 08:52:56,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 98f257e >2012-12-17 08:52:56,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fc49562 >2012-12-17 08:52:56,486 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:53:04,766 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:53:05,147 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [1a40d676] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:53:05,853 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:53:06,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d3bfd78 >2012-12-17 08:53:06,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76dc153 >2012-12-17 08:53:06,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:06,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, SpmStopVDSCommand, log id: 76dc153 >2012-12-17 08:53:06,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, ResetIrsVDSCommand, log id: 2d3bfd78 >2012-12-17 08:53:07,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 591d4c18 >2012-12-17 08:53:07,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:07,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 591d4c18 >2012-12-17 08:53:14,965 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:53:17,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:53:17,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4681 >2012-12-17 08:53:17,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4681, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 233d4423 >2012-12-17 08:53:17,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling started: taskId = 53859cae-6fd0-4895-ae39-c20cc465bd9c >2012-12-17 08:53:19,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling ended: taskId = 53859cae-6fd0-4895-ae39-c20cc465bd9c task status = finished >2012-12-17 08:53:19,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] spmStart polling ended, spm status: SPM >2012-12-17 08:53:19,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=53859cae-6fd0-4895-ae39-c20cc465bd9c), log id: 195dff7f >2012-12-17 08:53:19,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, HSMClearTaskVDSCommand, log id: 195dff7f >2012-12-17 08:53:19,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [32ec1ec3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a30f7db, log id: 233d4423 >2012-12-17 08:53:19,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:53:19,298 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:53:19,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d325152 >2012-12-17 08:53:19,332 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [32ec1ec3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:53:19,478 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [490300dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:53:19,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a4aac1 >2012-12-17 08:53:19,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f5ff814 >2012-12-17 08:53:19,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:19,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, SpmStopVDSCommand, log id: 2f5ff814 >2012-12-17 08:53:19,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, ResetIrsVDSCommand, log id: 3a4aac1 >2012-12-17 08:53:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6545a35 >2012-12-17 08:53:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6545a35 >2012-12-17 08:53:19,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:53:19,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4682 >2012-12-17 08:53:19,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4682, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2cf1dbac >2012-12-17 08:53:19,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = a8576c80-e37c-4742-8337-6e1f85043f30 >2012-12-17 08:53:20,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = a8576c80-e37c-4742-8337-6e1f85043f30 task status = finished >2012-12-17 08:53:20,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM >2012-12-17 08:53:20,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a8576c80-e37c-4742-8337-6e1f85043f30), log id: ba01558 >2012-12-17 08:53:20,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: ba01558 >2012-12-17 08:53:20,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ad94a5e, log id: 2cf1dbac >2012-12-17 08:53:21,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:53:21,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:21,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2693cda3 >2012-12-17 08:53:21,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2693cda3 >2012-12-17 08:53:21,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d325152 >2012-12-17 08:53:21,560 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:53:21,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31a19c1e >2012-12-17 08:53:21,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:21,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60c585fe >2012-12-17 08:53:21,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60c585fe >2012-12-17 08:53:21,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31a19c1e >2012-12-17 08:53:21,579 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:53:25,223 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:53:29,885 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [780c369f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:53:30,068 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [5bd08b5f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:53:30,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39c5503 >2012-12-17 08:53:30,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45635c70 >2012-12-17 08:53:30,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:31,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, SpmStopVDSCommand, log id: 45635c70 >2012-12-17 08:53:31,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, ResetIrsVDSCommand, log id: 39c5503 >2012-12-17 08:53:31,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4069f938 >2012-12-17 08:53:31,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5bd08b5f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:31,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5bd08b5f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4069f938 >2012-12-17 08:53:36,328 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:53:41,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:53:41,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4683 >2012-12-17 08:53:41,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4683, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 89419f >2012-12-17 08:53:41,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling started: taskId = 30d8b6fc-083c-451e-a99c-2f08b2142a03 >2012-12-17 08:53:42,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling ended: taskId = 30d8b6fc-083c-451e-a99c-2f08b2142a03 task status = finished >2012-12-17 08:53:42,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] spmStart polling ended, spm status: SPM >2012-12-17 08:53:42,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=30d8b6fc-083c-451e-a99c-2f08b2142a03), log id: a545803 >2012-12-17 08:53:42,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, HSMClearTaskVDSCommand, log id: a545803 >2012-12-17 08:53:42,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [337c1daa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@275e3241, log id: 89419f >2012-12-17 08:53:42,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:53:42,974 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:53:42,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7dc13231 >2012-12-17 08:53:42,992 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [337c1daa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:53:43,136 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [5a014659] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:53:43,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a981fcb >2012-12-17 08:53:43,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5502fa2d >2012-12-17 08:53:43,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:43,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, SpmStopVDSCommand, log id: 5502fa2d >2012-12-17 08:53:43,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, ResetIrsVDSCommand, log id: 4a981fcb >2012-12-17 08:53:43,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [5a014659] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6beef539 >2012-12-17 08:53:43,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:43,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [5a014659] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6beef539 >2012-12-17 08:53:43,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:53:43,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4684 >2012-12-17 08:53:43,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4684, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7090bab6 >2012-12-17 08:53:43,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling started: taskId = ebfc0b96-3694-42bf-a60c-f4c0043a02d5 >2012-12-17 08:53:44,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended: taskId = ebfc0b96-3694-42bf-a60c-f4c0043a02d5 task status = finished >2012-12-17 08:53:44,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended, spm status: SPM >2012-12-17 08:53:44,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ebfc0b96-3694-42bf-a60c-f4c0043a02d5), log id: 731baae7 >2012-12-17 08:53:44,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMClearTaskVDSCommand, log id: 731baae7 >2012-12-17 08:53:44,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d715ebb, log id: 7090bab6 >2012-12-17 08:53:45,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:53:45,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:45,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79176827 >2012-12-17 08:53:45,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79176827 >2012-12-17 08:53:45,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7dc13231 >2012-12-17 08:53:45,059 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:53:45,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fb7168c >2012-12-17 08:53:45,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:45,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ffaddd6 >2012-12-17 08:53:45,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ffaddd6 >2012-12-17 08:53:45,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fb7168c >2012-12-17 08:53:45,078 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [214f1ee4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:53:46,948 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:53:53,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6cc0b445] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:53:54,086 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [37630ba1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:53:54,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36970edb >2012-12-17 08:53:54,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21c010bb >2012-12-17 08:53:54,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:54,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, SpmStopVDSCommand, log id: 21c010bb >2012-12-17 08:53:54,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, ResetIrsVDSCommand, log id: 36970edb >2012-12-17 08:53:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 472ccf44 >2012-12-17 08:53:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:53:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [37630ba1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 472ccf44 >2012-12-17 08:53:57,156 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:54:04,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:04,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4685 >2012-12-17 08:54:04,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4685, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6dd47c44 >2012-12-17 08:54:04,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling started: taskId = f186d2ae-d377-444c-8a41-d4e92ff735d9 >2012-12-17 08:54:05,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling ended: taskId = f186d2ae-d377-444c-8a41-d4e92ff735d9 task status = finished >2012-12-17 08:54:06,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] spmStart polling ended, spm status: SPM >2012-12-17 08:54:06,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f186d2ae-d377-444c-8a41-d4e92ff735d9), log id: 751210b9 >2012-12-17 08:54:06,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, HSMClearTaskVDSCommand, log id: 751210b9 >2012-12-17 08:54:06,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [71e1da67] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75081dba, log id: 6dd47c44 >2012-12-17 08:54:06,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:06,673 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:54:06,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 109df0bf >2012-12-17 08:54:06,730 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [71e1da67] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:54:06,888 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:54:06,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4da03343 >2012-12-17 08:54:06,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 29604dbf >2012-12-17 08:54:06,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:07,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SpmStopVDSCommand, log id: 29604dbf >2012-12-17 08:54:07,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, ResetIrsVDSCommand, log id: 4da03343 >2012-12-17 08:54:07,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: acaa2c6 >2012-12-17 08:54:07,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:07,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: acaa2c6 >2012-12-17 08:54:07,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:07,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4686 >2012-12-17 08:54:07,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4686, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d4cb4aa >2012-12-17 08:54:07,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling started: taskId = 02d9e6de-f0e5-4992-a5df-5b33b617e1bd >2012-12-17 08:54:08,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling ended: taskId = 02d9e6de-f0e5-4992-a5df-5b33b617e1bd task status = finished >2012-12-17 08:54:08,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] spmStart polling ended, spm status: SPM >2012-12-17 08:54:08,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02d9e6de-f0e5-4992-a5df-5b33b617e1bd), log id: 6b9ed15a >2012-12-17 08:54:08,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMClearTaskVDSCommand, log id: 6b9ed15a >2012-12-17 08:54:08,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c2a2e9, log id: 6d4cb4aa >2012-12-17 08:54:08,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:08,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:08,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 496c5c1e >2012-12-17 08:54:08,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 496c5c1e >2012-12-17 08:54:08,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 109df0bf >2012-12-17 08:54:08,372 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:08,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 169df2bb >2012-12-17 08:54:08,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:08,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 556ea340 >2012-12-17 08:54:08,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 556ea340 >2012-12-17 08:54:08,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 169df2bb >2012-12-17 08:54:08,391 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [3e3e3ebc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:17,250 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4232612d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:54:17,432 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [79e50d62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:54:17,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a4335 >2012-12-17 08:54:17,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f94e7cc >2012-12-17 08:54:17,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:17,598 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:54:18,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, SpmStopVDSCommand, log id: 2f94e7cc >2012-12-17 08:54:18,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, ResetIrsVDSCommand, log id: 6a4335 >2012-12-17 08:54:18,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31ea2f20 >2012-12-17 08:54:18,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [79e50d62] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:18,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [79e50d62] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31ea2f20 >2012-12-17 08:54:28,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:28,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4687 >2012-12-17 08:54:28,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4687, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37e53615 >2012-12-17 08:54:28,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling started: taskId = 6cb8e301-8614-4f2e-95c5-a31588ff371f >2012-12-17 08:54:29,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling ended: taskId = 6cb8e301-8614-4f2e-95c5-a31588ff371f task status = finished >2012-12-17 08:54:29,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] spmStart polling ended, spm status: SPM >2012-12-17 08:54:29,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [d00852e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6cb8e301-8614-4f2e-95c5-a31588ff371f), log id: 5d608511 >2012-12-17 08:54:29,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, HSMClearTaskVDSCommand, log id: 5d608511 >2012-12-17 08:54:29,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [d00852e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33908c84, log id: 37e53615 >2012-12-17 08:54:29,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:29,484 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:54:29,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e595dac >2012-12-17 08:54:29,501 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [d00852e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:54:29,706 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [15510694] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:54:29,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2888804e >2012-12-17 08:54:29,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a9ab0ae >2012-12-17 08:54:29,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:30,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, SpmStopVDSCommand, log id: 7a9ab0ae >2012-12-17 08:54:30,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, ResetIrsVDSCommand, log id: 2888804e >2012-12-17 08:54:30,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4514948e >2012-12-17 08:54:30,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:30,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4514948e >2012-12-17 08:54:30,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:30,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4688 >2012-12-17 08:54:30,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4688, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5692ab98 >2012-12-17 08:54:30,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = e2ef5666-365a-4fac-8a08-89fcfd4b0a70 >2012-12-17 08:54:31,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = e2ef5666-365a-4fac-8a08-89fcfd4b0a70 task status = finished >2012-12-17 08:54:31,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM >2012-12-17 08:54:31,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e2ef5666-365a-4fac-8a08-89fcfd4b0a70), log id: 21a78d6 >2012-12-17 08:54:31,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 21a78d6 >2012-12-17 08:54:31,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17f32aae, log id: 5692ab98 >2012-12-17 08:54:31,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:31,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:31,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5a1dfe90 >2012-12-17 08:54:31,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5a1dfe90 >2012-12-17 08:54:31,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e595dac >2012-12-17 08:54:31,248 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:31,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6733ce28 >2012-12-17 08:54:31,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:31,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 98ffb2c >2012-12-17 08:54:31,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 98ffb2c >2012-12-17 08:54:31,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6733ce28 >2012-12-17 08:54:31,267 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [ccd5055] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:37,987 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:54:40,153 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c95e244] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:54:40,458 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [192de1a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:54:41,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6d27354c >2012-12-17 08:54:41,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51b2c770 >2012-12-17 08:54:41,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:41,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, SpmStopVDSCommand, log id: 51b2c770 >2012-12-17 08:54:41,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, ResetIrsVDSCommand, log id: 6d27354c >2012-12-17 08:54:41,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ac82626 >2012-12-17 08:54:41,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:41,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ac82626 >2012-12-17 08:54:48,182 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:54:51,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:51,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4689 >2012-12-17 08:54:51,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4689, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3efe533d >2012-12-17 08:54:51,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling started: taskId = 91a11e87-ce54-4091-832c-f5e1a8f6c1cc >2012-12-17 08:54:52,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended: taskId = 91a11e87-ce54-4091-832c-f5e1a8f6c1cc task status = finished >2012-12-17 08:54:52,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] spmStart polling ended, spm status: SPM >2012-12-17 08:54:52,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=91a11e87-ce54-4091-832c-f5e1a8f6c1cc), log id: 2afb0a2b >2012-12-17 08:54:52,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, HSMClearTaskVDSCommand, log id: 2afb0a2b >2012-12-17 08:54:52,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [50f32e72] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4758a318, log id: 3efe533d >2012-12-17 08:54:52,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:52,803 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:54:52,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8da5c0 >2012-12-17 08:54:52,837 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [50f32e72] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:54:52,947 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [46a80193] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:54:52,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e581718 >2012-12-17 08:54:52,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 70b3167d >2012-12-17 08:54:53,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:53,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SpmStopVDSCommand, log id: 70b3167d >2012-12-17 08:54:53,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, ResetIrsVDSCommand, log id: 3e581718 >2012-12-17 08:54:53,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ba5d24f >2012-12-17 08:54:53,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:53,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ba5d24f >2012-12-17 08:54:53,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:54:53,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4690 >2012-12-17 08:54:53,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4690, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79eb4118 >2012-12-17 08:54:53,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = a3978184-ca31-4d15-bff9-a4e7512b5813 >2012-12-17 08:54:54,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = a3978184-ca31-4d15-bff9-a4e7512b5813 task status = finished >2012-12-17 08:54:54,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM >2012-12-17 08:54:54,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3978184-ca31-4d15-bff9-a4e7512b5813), log id: 8e5f177 >2012-12-17 08:54:54,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 8e5f177 >2012-12-17 08:54:54,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@304fc031, log id: 79eb4118 >2012-12-17 08:54:54,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:54:54,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:54,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: aac74de >2012-12-17 08:54:54,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: aac74de >2012-12-17 08:54:55,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8da5c0 >2012-12-17 08:54:55,000 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:55,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 402ca04b >2012-12-17 08:54:55,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:54:55,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 272c4b70 >2012-12-17 08:54:55,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 272c4b70 >2012-12-17 08:54:55,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 402ca04b >2012-12-17 08:54:55,020 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [7d79337c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:54:58,401 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Checking autorecoverable hosts >2012-12-17 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Autorecovering 0 hosts >2012-12-17 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Checking autorecoverable hosts done >2012-12-17 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [419474a0] Autorecovering storage domains is disabled, skipping >2012-12-17 08:55:03,918 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [79e50d62] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:55:04,043 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [4dd32292] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:55:04,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4aa0d827 >2012-12-17 08:55:04,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2afb42d9 >2012-12-17 08:55:04,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:04,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStopVDSCommand, log id: 2afb42d9 >2012-12-17 08:55:04,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, ResetIrsVDSCommand, log id: 4aa0d827 >2012-12-17 08:55:04,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d4f1e7f >2012-12-17 08:55:04,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:04,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d4f1e7f >2012-12-17 08:55:08,600 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:55:14,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:55:14,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4691 >2012-12-17 08:55:14,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4691, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f7ed5fe >2012-12-17 08:55:14,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling started: taskId = 4b479d6a-8970-4261-a0a2-39543927e7a1 >2012-12-17 08:55:15,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling ended: taskId = 4b479d6a-8970-4261-a0a2-39543927e7a1 task status = finished >2012-12-17 08:55:15,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] spmStart polling ended, spm status: SPM >2012-12-17 08:55:15,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b479d6a-8970-4261-a0a2-39543927e7a1), log id: 7aa7c379 >2012-12-17 08:55:15,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, HSMClearTaskVDSCommand, log id: 7aa7c379 >2012-12-17 08:55:15,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2ba7b70d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c127bd8, log id: 5f7ed5fe >2012-12-17 08:55:15,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:55:16,142 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:55:16,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 621299fa >2012-12-17 08:55:16,157 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2ba7b70d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:55:16,291 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [4256c72b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:55:16,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a2f2df7 >2012-12-17 08:55:16,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4687aeaf >2012-12-17 08:55:16,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:16,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, SpmStopVDSCommand, log id: 4687aeaf >2012-12-17 08:55:16,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, ResetIrsVDSCommand, log id: 3a2f2df7 >2012-12-17 08:55:16,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b094d46 >2012-12-17 08:55:16,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:16,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4256c72b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b094d46 >2012-12-17 08:55:16,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:55:16,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4692 >2012-12-17 08:55:16,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4692, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5eceafa >2012-12-17 08:55:16,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling started: taskId = 4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76 >2012-12-17 08:55:17,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling ended: taskId = 4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76 task status = finished >2012-12-17 08:55:17,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] spmStart polling ended, spm status: SPM >2012-12-17 08:55:17,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d5ca2be-40f2-4c97-83c4-a84a1bcb2f76), log id: 2a8ff032 >2012-12-17 08:55:17,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMClearTaskVDSCommand, log id: 2a8ff032 >2012-12-17 08:55:17,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4fe27872, log id: 5eceafa >2012-12-17 08:55:17,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:55:17,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:17,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4292e292 >2012-12-17 08:55:17,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4292e292 >2012-12-17 08:55:17,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 621299fa >2012-12-17 08:55:17,810 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:55:17,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39590163 >2012-12-17 08:55:17,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:17,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 197edd7 >2012-12-17 08:55:17,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 197edd7 >2012-12-17 08:55:17,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [3d206f3c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39590163 >2012-12-17 08:55:17,830 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [3d206f3c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:55:18,888 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:55:26,707 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2f94bcb4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:55:26,878 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [1407e3a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:55:26,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 488bdd79 >2012-12-17 08:55:26,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44968b88 >2012-12-17 08:55:26,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:27,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStopVDSCommand, log id: 44968b88 >2012-12-17 08:55:27,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, ResetIrsVDSCommand, log id: 488bdd79 >2012-12-17 08:55:27,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c458755 >2012-12-17 08:55:27,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:27,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c458755 >2012-12-17 08:55:29,101 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:55:37,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:55:37,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4693 >2012-12-17 08:55:37,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4693, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50813ea2 >2012-12-17 08:55:37,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling started: taskId = 0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6 >2012-12-17 08:55:38,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling ended: taskId = 0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6 task status = finished >2012-12-17 08:55:38,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] spmStart polling ended, spm status: SPM >2012-12-17 08:55:38,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [34599a45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0ad7c0fc-a7c6-42bf-bb79-dff716f6a8d6), log id: 17dc15a1 >2012-12-17 08:55:38,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, HSMClearTaskVDSCommand, log id: 17dc15a1 >2012-12-17 08:55:38,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [34599a45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72100943, log id: 50813ea2 >2012-12-17 08:55:38,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:55:38,403 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:55:38,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c854a17 >2012-12-17 08:55:38,421 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [34599a45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:55:38,546 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [6cb72173] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:55:38,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17237abd >2012-12-17 08:55:38,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5bbbd5b7 >2012-12-17 08:55:38,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:38,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, SpmStopVDSCommand, log id: 5bbbd5b7 >2012-12-17 08:55:38,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, ResetIrsVDSCommand, log id: 17237abd >2012-12-17 08:55:38,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61076859 >2012-12-17 08:55:38,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6cb72173] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:38,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [6cb72173] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61076859 >2012-12-17 08:55:38,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:55:38,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4694 >2012-12-17 08:55:38,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4694, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 557276fb >2012-12-17 08:55:38,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 8f8f7993-5b75-4af6-b7c0-143aa38adf73 >2012-12-17 08:55:39,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 8f8f7993-5b75-4af6-b7c0-143aa38adf73 task status = finished >2012-12-17 08:55:39,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 08:55:39,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8f8f7993-5b75-4af6-b7c0-143aa38adf73), log id: 404d51f7 >2012-12-17 08:55:40,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 404d51f7 >2012-12-17 08:55:40,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1440501c, log id: 557276fb >2012-12-17 08:55:40,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:55:40,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:40,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e168dc7 >2012-12-17 08:55:40,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e168dc7 >2012-12-17 08:55:40,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c854a17 >2012-12-17 08:55:40,046 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:55:40,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3014d918 >2012-12-17 08:55:40,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:40,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 123f2d66 >2012-12-17 08:55:40,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 123f2d66 >2012-12-17 08:55:40,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3014d918 >2012-12-17 08:55:40,066 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:55:48,896 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [dd34b5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:55:48,997 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [48c4b2a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:55:49,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5480ff24 >2012-12-17 08:55:49,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5b64ba6b >2012-12-17 08:55:49,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:49,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, SpmStopVDSCommand, log id: 5b64ba6b >2012-12-17 08:55:49,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, ResetIrsVDSCommand, log id: 5480ff24 >2012-12-17 08:55:49,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 721c04e2 >2012-12-17 08:55:49,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [48c4b2a7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:55:49,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [48c4b2a7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 721c04e2 >2012-12-17 08:55:49,564 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:55:59,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:55:59,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4695 >2012-12-17 08:55:59,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4695, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 468d743 >2012-12-17 08:55:59,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling started: taskId = 020bf939-3f26-4c94-afcb-7cc000ec49f5 >2012-12-17 08:56:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling ended: taskId = 020bf939-3f26-4c94-afcb-7cc000ec49f5 task status = finished >2012-12-17 08:56:00,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] spmStart polling ended, spm status: SPM >2012-12-17 08:56:00,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [45590d60] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=020bf939-3f26-4c94-afcb-7cc000ec49f5), log id: 2e0139d1 >2012-12-17 08:56:00,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, HSMClearTaskVDSCommand, log id: 2e0139d1 >2012-12-17 08:56:00,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [45590d60] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@527adfef, log id: 468d743 >2012-12-17 08:56:00,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:00,953 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:56:00,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15192f89 >2012-12-17 08:56:01,066 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [45590d60] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:01,381 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:01,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 153c787a >2012-12-17 08:56:01,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 455860cc >2012-12-17 08:56:01,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:01,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, SpmStopVDSCommand, log id: 455860cc >2012-12-17 08:56:01,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, ResetIrsVDSCommand, log id: 153c787a >2012-12-17 08:56:01,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1599796e >2012-12-17 08:56:01,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:01,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [4c00bcb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1599796e >2012-12-17 08:56:01,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:56:02,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4696 >2012-12-17 08:56:02,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4696, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f4d187d >2012-12-17 08:56:02,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = 81716757-3b28-4230-85c3-2aaf9e86adcc >2012-12-17 08:56:03,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = 81716757-3b28-4230-85c3-2aaf9e86adcc task status = finished >2012-12-17 08:56:03,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM >2012-12-17 08:56:03,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81716757-3b28-4230-85c3-2aaf9e86adcc), log id: 1c0d4bc8 >2012-12-17 08:56:03,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 1c0d4bc8 >2012-12-17 08:56:03,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2cea45db, log id: 1f4d187d >2012-12-17 08:56:03,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:03,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:03,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dfdb312 >2012-12-17 08:56:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dfdb312 >2012-12-17 08:56:03,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15192f89 >2012-12-17 08:56:03,178 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:03,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: af769eb >2012-12-17 08:56:03,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:03,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7365931a >2012-12-17 08:56:03,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7365931a >2012-12-17 08:56:03,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: af769eb >2012-12-17 08:56:03,198 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [7b0d0943] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:10,079 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:56:12,021 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5266d504] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:12,135 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [46e0c818] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:12,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 74f44789 >2012-12-17 08:56:12,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26944008 >2012-12-17 08:56:12,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:12,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, SpmStopVDSCommand, log id: 26944008 >2012-12-17 08:56:12,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, ResetIrsVDSCommand, log id: 74f44789 >2012-12-17 08:56:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79d20ae5 >2012-12-17 08:56:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79d20ae5 >2012-12-17 08:56:20,313 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:56:22,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:56:22,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4697 >2012-12-17 08:56:22,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4697, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c6b8b1 >2012-12-17 08:56:22,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling started: taskId = 9e48c035-7f0d-4f29-9aac-dffb46204705 >2012-12-17 08:56:23,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling ended: taskId = 9e48c035-7f0d-4f29-9aac-dffb46204705 task status = finished >2012-12-17 08:56:23,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] spmStart polling ended, spm status: SPM >2012-12-17 08:56:23,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9e48c035-7f0d-4f29-9aac-dffb46204705), log id: 4af6fe4b >2012-12-17 08:56:23,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, HSMClearTaskVDSCommand, log id: 4af6fe4b >2012-12-17 08:56:23,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [44f1dfe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1db11460, log id: c6b8b1 >2012-12-17 08:56:23,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:23,610 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:56:23,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e1065b5 >2012-12-17 08:56:23,650 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [44f1dfe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:23,798 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [6fcdea84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:23,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bdfc0d8 >2012-12-17 08:56:23,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7fb1e469 >2012-12-17 08:56:23,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:24,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, SpmStopVDSCommand, log id: 7fb1e469 >2012-12-17 08:56:24,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, ResetIrsVDSCommand, log id: 2bdfc0d8 >2012-12-17 08:56:24,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6bb4df1d >2012-12-17 08:56:24,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6fcdea84] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:24,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6fcdea84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6bb4df1d >2012-12-17 08:56:24,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:56:24,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4698 >2012-12-17 08:56:24,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4698, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66ff6b15 >2012-12-17 08:56:24,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling started: taskId = c84f2dff-3b62-4c08-b300-074036761678 >2012-12-17 08:56:25,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended: taskId = c84f2dff-3b62-4c08-b300-074036761678 task status = finished >2012-12-17 08:56:25,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended, spm status: SPM >2012-12-17 08:56:25,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c84f2dff-3b62-4c08-b300-074036761678), log id: 37ae0b44 >2012-12-17 08:56:25,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMClearTaskVDSCommand, log id: 37ae0b44 >2012-12-17 08:56:25,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@365b9eb, log id: 66ff6b15 >2012-12-17 08:56:25,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:25,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:25,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 206f0c74 >2012-12-17 08:56:25,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 206f0c74 >2012-12-17 08:56:25,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e1065b5 >2012-12-17 08:56:25,325 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:25,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a9aa44e >2012-12-17 08:56:25,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:25,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e6576fd >2012-12-17 08:56:25,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e6576fd >2012-12-17 08:56:25,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a9aa44e >2012-12-17 08:56:25,346 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [46701658] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:31,256 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:56:34,175 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1bff0c70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:34,352 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [67ce2e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:34,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68010d29 >2012-12-17 08:56:34,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7c2feba >2012-12-17 08:56:34,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:34,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, SpmStopVDSCommand, log id: 7c2feba >2012-12-17 08:56:34,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, ResetIrsVDSCommand, log id: 68010d29 >2012-12-17 08:56:34,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c28a75f >2012-12-17 08:56:34,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:34,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c28a75f >2012-12-17 08:56:41,481 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:56:44,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:56:44,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4699 >2012-12-17 08:56:44,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4699, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 533c6115 >2012-12-17 08:56:44,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling started: taskId = 6b9cc4bf-8854-4ade-895b-de419f384d0e >2012-12-17 08:56:45,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling ended: taskId = 6b9cc4bf-8854-4ade-895b-de419f384d0e task status = finished >2012-12-17 08:56:45,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] spmStart polling ended, spm status: SPM >2012-12-17 08:56:45,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b9cc4bf-8854-4ade-895b-de419f384d0e), log id: 493dc77c >2012-12-17 08:56:45,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, HSMClearTaskVDSCommand, log id: 493dc77c >2012-12-17 08:56:45,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [49c4851d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76f3c374, log id: 533c6115 >2012-12-17 08:56:46,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:46,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:56:46,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2888347 >2012-12-17 08:56:46,464 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [49c4851d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:46,762 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [d0709d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:46,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32d038ee >2012-12-17 08:56:46,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bdb6b00 >2012-12-17 08:56:46,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:46,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SpmStopVDSCommand, log id: 6bdb6b00 >2012-12-17 08:56:47,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, ResetIrsVDSCommand, log id: 32d038ee >2012-12-17 08:56:47,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5549cb05 >2012-12-17 08:56:47,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:47,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5549cb05 >2012-12-17 08:56:47,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:56:47,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4700 >2012-12-17 08:56:47,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4700, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f463b21 >2012-12-17 08:56:47,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = 81baedd9-cb25-4130-a2d0-89e3980bcdb7 >2012-12-17 08:56:48,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = 81baedd9-cb25-4130-a2d0-89e3980bcdb7 task status = finished >2012-12-17 08:56:48,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM >2012-12-17 08:56:48,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81baedd9-cb25-4130-a2d0-89e3980bcdb7), log id: 3072e91b >2012-12-17 08:56:48,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 3072e91b >2012-12-17 08:56:48,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48260476, log id: f463b21 >2012-12-17 08:56:48,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:56:48,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:48,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5add7ddd >2012-12-17 08:56:48,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5add7ddd >2012-12-17 08:56:48,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2888347 >2012-12-17 08:56:48,281 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:48,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4bb952ee >2012-12-17 08:56:48,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:48,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3900e32 >2012-12-17 08:56:48,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3900e32 >2012-12-17 08:56:48,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4bb952ee >2012-12-17 08:56:48,301 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:56:51,705 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:56:57,154 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [7c1d0790] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:56:57,297 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [356a6c1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:56:57,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 212ba09c >2012-12-17 08:56:57,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bc5a3bf >2012-12-17 08:56:57,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:58,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, SpmStopVDSCommand, log id: 6bc5a3bf >2012-12-17 08:56:58,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, ResetIrsVDSCommand, log id: 212ba09c >2012-12-17 08:56:58,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22aec287 >2012-12-17 08:56:58,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:56:58,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22aec287 >2012-12-17 08:57:01,922 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:57:08,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:08,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4701 >2012-12-17 08:57:08,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4701, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5974d99a >2012-12-17 08:57:08,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling started: taskId = 71c8f917-b832-41a0-8d44-0747cc4b0e42 >2012-12-17 08:57:09,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling ended: taskId = 71c8f917-b832-41a0-8d44-0747cc4b0e42 task status = finished >2012-12-17 08:57:09,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] spmStart polling ended, spm status: SPM >2012-12-17 08:57:09,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [44e11955] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=71c8f917-b832-41a0-8d44-0747cc4b0e42), log id: 66a804f6 >2012-12-17 08:57:09,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, HSMClearTaskVDSCommand, log id: 66a804f6 >2012-12-17 08:57:09,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [44e11955] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20dde721, log id: 5974d99a >2012-12-17 08:57:09,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:09,517 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:57:09,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 442c314d >2012-12-17 08:57:09,548 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [44e11955] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:57:09,713 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [1925ecb1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:57:09,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2103655f >2012-12-17 08:57:09,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a6e9afa >2012-12-17 08:57:09,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:10,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStopVDSCommand, log id: 7a6e9afa >2012-12-17 08:57:11,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, ResetIrsVDSCommand, log id: 2103655f >2012-12-17 08:57:11,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5acb1f70 >2012-12-17 08:57:11,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:11,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5acb1f70 >2012-12-17 08:57:11,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:11,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4702 >2012-12-17 08:57:11,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4702, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fd6a72b >2012-12-17 08:57:11,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling started: taskId = c3c21c65-5983-42b0-8d19-984b0b34ba3d >2012-12-17 08:57:12,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling ended: taskId = c3c21c65-5983-42b0-8d19-984b0b34ba3d task status = finished >2012-12-17 08:57:12,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] spmStart polling ended, spm status: SPM >2012-12-17 08:57:12,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c3c21c65-5983-42b0-8d19-984b0b34ba3d), log id: 18b267ae >2012-12-17 08:57:12,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMClearTaskVDSCommand, log id: 18b267ae >2012-12-17 08:57:12,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68b4dd2c, log id: 2fd6a72b >2012-12-17 08:57:12,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:12,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:12,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f0159bc >2012-12-17 08:57:12,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f0159bc >2012-12-17 08:57:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 442c314d >2012-12-17 08:57:12,392 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:57:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a0bb078 >2012-12-17 08:57:12,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:12,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27704a28 >2012-12-17 08:57:12,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27704a28 >2012-12-17 08:57:12,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [6d961413] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a0bb078 >2012-12-17 08:57:12,413 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [6d961413] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:57:21,292 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [48f6cd0b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:57:21,821 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [33e6033] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:57:21,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 46946f56 >2012-12-17 08:57:21,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e3e7723 >2012-12-17 08:57:21,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:22,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, SpmStopVDSCommand, log id: 1e3e7723 >2012-12-17 08:57:22,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, ResetIrsVDSCommand, log id: 46946f56 >2012-12-17 08:57:22,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [33e6033] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51880028 >2012-12-17 08:57:22,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [33e6033] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:22,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [33e6033] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51880028 >2012-12-17 08:57:22,358 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:57:32,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:32,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4703 >2012-12-17 08:57:32,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4703, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c85f4c2 >2012-12-17 08:57:32,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling started: taskId = 9f54f88b-beea-4f1c-b4ab-5c6d162285b8 >2012-12-17 08:57:33,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling ended: taskId = 9f54f88b-beea-4f1c-b4ab-5c6d162285b8 task status = finished >2012-12-17 08:57:33,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] spmStart polling ended, spm status: SPM >2012-12-17 08:57:33,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [4793062d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9f54f88b-beea-4f1c-b4ab-5c6d162285b8), log id: 752d01a5 >2012-12-17 08:57:33,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, HSMClearTaskVDSCommand, log id: 752d01a5 >2012-12-17 08:57:33,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [4793062d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5dc733bc, log id: 5c85f4c2 >2012-12-17 08:57:33,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:33,352 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:57:33,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 184b53f6 >2012-12-17 08:57:33,363 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [4793062d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:57:33,518 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [49b46a39] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:57:33,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21972ad6 >2012-12-17 08:57:33,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68f2910e >2012-12-17 08:57:33,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:34,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, SpmStopVDSCommand, log id: 68f2910e >2012-12-17 08:57:34,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, ResetIrsVDSCommand, log id: 21972ad6 >2012-12-17 08:57:34,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39256663 >2012-12-17 08:57:34,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [49b46a39] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:34,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [49b46a39] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39256663 >2012-12-17 08:57:34,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:34,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4704 >2012-12-17 08:57:34,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4704, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bbd10cd >2012-12-17 08:57:34,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling started: taskId = b9e22d9e-606c-4671-a324-cf9bc3bbe76c >2012-12-17 08:57:35,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling ended: taskId = b9e22d9e-606c-4671-a324-cf9bc3bbe76c task status = finished >2012-12-17 08:57:35,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] spmStart polling ended, spm status: SPM >2012-12-17 08:57:35,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b9e22d9e-606c-4671-a324-cf9bc3bbe76c), log id: 4607e6f5 >2012-12-17 08:57:35,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMClearTaskVDSCommand, log id: 4607e6f5 >2012-12-17 08:57:35,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30cc02be, log id: 6bbd10cd >2012-12-17 08:57:36,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:36,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:36,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31f9e0fb >2012-12-17 08:57:36,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31f9e0fb >2012-12-17 08:57:36,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 184b53f6 >2012-12-17 08:57:36,223 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:57:36,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 719a8b31 >2012-12-17 08:57:36,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:36,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 134a5c26 >2012-12-17 08:57:36,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 134a5c26 >2012-12-17 08:57:36,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [d0709d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 719a8b31 >2012-12-17 08:57:36,244 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [d0709d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:57:42,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:57:44,664 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [3cf5d2fd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:57:44,820 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [346a0820] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:57:44,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c7f8fbb >2012-12-17 08:57:44,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4e313ef6 >2012-12-17 08:57:44,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:45,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStopVDSCommand, log id: 4e313ef6 >2012-12-17 08:57:45,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, ResetIrsVDSCommand, log id: 6c7f8fbb >2012-12-17 08:57:45,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6290a559 >2012-12-17 08:57:45,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:45,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6290a559 >2012-12-17 08:57:53,085 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:57:55,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:55,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4705 >2012-12-17 08:57:55,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4705, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 328b6a4c >2012-12-17 08:57:55,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling started: taskId = 5ca699a5-9352-4817-9c23-cb898ac3dc67 >2012-12-17 08:57:56,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling ended: taskId = 5ca699a5-9352-4817-9c23-cb898ac3dc67 task status = finished >2012-12-17 08:57:56,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] spmStart polling ended, spm status: SPM >2012-12-17 08:57:56,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ca699a5-9352-4817-9c23-cb898ac3dc67), log id: 68b5d11d >2012-12-17 08:57:56,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, HSMClearTaskVDSCommand, log id: 68b5d11d >2012-12-17 08:57:56,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [3969aa82] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b3fc9f, log id: 328b6a4c >2012-12-17 08:57:56,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:56,620 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:57:56,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b8228fe >2012-12-17 08:57:56,656 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3969aa82] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:57:56,783 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [278943a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:57:56,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ef1926a >2012-12-17 08:57:56,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 742c0266 >2012-12-17 08:57:56,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:57,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, SpmStopVDSCommand, log id: 742c0266 >2012-12-17 08:57:57,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, ResetIrsVDSCommand, log id: 2ef1926a >2012-12-17 08:57:57,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [278943a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 749c520e >2012-12-17 08:57:57,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [278943a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:57,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [278943a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 749c520e >2012-12-17 08:57:57,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:57:57,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4706 >2012-12-17 08:57:57,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4706, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 402a76c8 >2012-12-17 08:57:57,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling started: taskId = 0b5da0be-b423-418a-9dbf-65414047a66b >2012-12-17 08:57:58,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended: taskId = 0b5da0be-b423-418a-9dbf-65414047a66b task status = finished >2012-12-17 08:57:58,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] spmStart polling ended, spm status: SPM >2012-12-17 08:57:58,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0b5da0be-b423-418a-9dbf-65414047a66b), log id: 3bee9e6b >2012-12-17 08:57:58,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMClearTaskVDSCommand, log id: 3bee9e6b >2012-12-17 08:57:58,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ee2a18a, log id: 402a76c8 >2012-12-17 08:57:58,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:57:59,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:59,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a42e8f >2012-12-17 08:57:59,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a42e8f >2012-12-17 08:57:59,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b8228fe >2012-12-17 08:57:59,037 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:57:59,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40a78a73 >2012-12-17 08:57:59,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:57:59,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fa4347e >2012-12-17 08:57:59,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fa4347e >2012-12-17 08:57:59,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [3e5a5a03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40a78a73 >2012-12-17 08:57:59,058 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [3e5a5a03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:58:03,364 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:58:07,910 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1a6bc63] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:58:08,035 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [38fb24e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:58:08,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 398e693a >2012-12-17 08:58:08,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18d3d562 >2012-12-17 08:58:08,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:08,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, SpmStopVDSCommand, log id: 18d3d562 >2012-12-17 08:58:08,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, ResetIrsVDSCommand, log id: 398e693a >2012-12-17 08:58:08,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34b3ad23 >2012-12-17 08:58:08,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:08,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34b3ad23 >2012-12-17 08:58:13,728 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:58:18,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:58:18,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4707 >2012-12-17 08:58:18,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4707, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3aa903b6 >2012-12-17 08:58:18,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling started: taskId = 35c9239e-f779-41b2-bf0b-56599500ef00 >2012-12-17 08:58:19,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling ended: taskId = 35c9239e-f779-41b2-bf0b-56599500ef00 task status = finished >2012-12-17 08:58:19,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] spmStart polling ended, spm status: SPM >2012-12-17 08:58:19,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [1390e512] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35c9239e-f779-41b2-bf0b-56599500ef00), log id: 372319bb >2012-12-17 08:58:19,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, HSMClearTaskVDSCommand, log id: 372319bb >2012-12-17 08:58:19,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [1390e512] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@616e6044, log id: 3aa903b6 >2012-12-17 08:58:19,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:58:19,684 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:58:19,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c3e0f43 >2012-12-17 08:58:19,779 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [1390e512] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:58:20,014 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [22fa8e65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:58:20,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3114a173 >2012-12-17 08:58:20,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 510b3404 >2012-12-17 08:58:20,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:20,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SpmStopVDSCommand, log id: 510b3404 >2012-12-17 08:58:20,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, ResetIrsVDSCommand, log id: 3114a173 >2012-12-17 08:58:20,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e0eecac >2012-12-17 08:58:20,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:20,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e0eecac >2012-12-17 08:58:20,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:58:20,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4708 >2012-12-17 08:58:20,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4708, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 669ada70 >2012-12-17 08:58:20,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling started: taskId = a6aae032-6170-4656-ae8a-a9e354b52342 >2012-12-17 08:58:22,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended: taskId = a6aae032-6170-4656-ae8a-a9e354b52342 task status = finished >2012-12-17 08:58:22,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended, spm status: SPM >2012-12-17 08:58:22,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6aae032-6170-4656-ae8a-a9e354b52342), log id: 13a75b16 >2012-12-17 08:58:22,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMClearTaskVDSCommand, log id: 13a75b16 >2012-12-17 08:58:22,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@504d0b6e, log id: 669ada70 >2012-12-17 08:58:22,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:58:22,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:22,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7aec2e08 >2012-12-17 08:58:22,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7aec2e08 >2012-12-17 08:58:22,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c3e0f43 >2012-12-17 08:58:22,139 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:58:22,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 723155fe >2012-12-17 08:58:22,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:22,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1eba62be >2012-12-17 08:58:22,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1eba62be >2012-12-17 08:58:22,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 723155fe >2012-12-17 08:58:22,160 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [446aaffb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:58:23,922 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:58:30,836 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5571991a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:58:31,125 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [67d5cb38] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:58:31,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 678bb11a >2012-12-17 08:58:31,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f3a9a95 >2012-12-17 08:58:31,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:31,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, SpmStopVDSCommand, log id: 6f3a9a95 >2012-12-17 08:58:31,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, ResetIrsVDSCommand, log id: 678bb11a >2012-12-17 08:58:31,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b678a66 >2012-12-17 08:58:31,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:31,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b678a66 >2012-12-17 08:58:34,111 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:58:41,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:58:41,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4709 >2012-12-17 08:58:41,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4709, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39265ae9 >2012-12-17 08:58:41,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling started: taskId = 9c0f610f-ff10-4028-865c-179a22ae4985 >2012-12-17 08:58:42,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling ended: taskId = 9c0f610f-ff10-4028-865c-179a22ae4985 task status = finished >2012-12-17 08:58:42,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] spmStart polling ended, spm status: SPM >2012-12-17 08:58:42,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c0f610f-ff10-4028-865c-179a22ae4985), log id: 1b8f4e41 >2012-12-17 08:58:42,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, HSMClearTaskVDSCommand, log id: 1b8f4e41 >2012-12-17 08:58:42,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [192de1a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5a9b89, log id: 39265ae9 >2012-12-17 08:58:42,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:58:42,827 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:58:42,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b71085f >2012-12-17 08:58:42,874 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [192de1a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:58:42,976 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [5044c757] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:58:43,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2986d40e >2012-12-17 08:58:43,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b554e9f >2012-12-17 08:58:43,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:43,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SpmStopVDSCommand, log id: 4b554e9f >2012-12-17 08:58:43,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, ResetIrsVDSCommand, log id: 2986d40e >2012-12-17 08:58:43,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26a5e912 >2012-12-17 08:58:43,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:43,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26a5e912 >2012-12-17 08:58:43,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:58:43,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4710 >2012-12-17 08:58:43,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4710, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44422c2d >2012-12-17 08:58:43,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling started: taskId = f16d4c87-3821-4fc9-aa0d-a7f07d9fb534 >2012-12-17 08:58:44,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended: taskId = f16d4c87-3821-4fc9-aa0d-a7f07d9fb534 task status = finished >2012-12-17 08:58:44,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] spmStart polling ended, spm status: SPM >2012-12-17 08:58:44,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f16d4c87-3821-4fc9-aa0d-a7f07d9fb534), log id: 386fc26b >2012-12-17 08:58:44,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMClearTaskVDSCommand, log id: 386fc26b >2012-12-17 08:58:44,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31451bbf, log id: 44422c2d >2012-12-17 08:58:44,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:58:44,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:44,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13b4169f >2012-12-17 08:58:44,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13b4169f >2012-12-17 08:58:44,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b71085f >2012-12-17 08:58:44,984 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:58:44,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 352aceab >2012-12-17 08:58:44,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:44,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 151549af >2012-12-17 08:58:45,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 151549af >2012-12-17 08:58:45,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-52) [65048227] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 352aceab >2012-12-17 08:58:45,003 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-52) [65048227] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:58:53,859 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5ad4c70] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:58:54,040 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [441eec2e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:58:54,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e9ccd8a >2012-12-17 08:58:54,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 767d2c88 >2012-12-17 08:58:54,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:54,483 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:58:54,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStopVDSCommand, log id: 767d2c88 >2012-12-17 08:58:54,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, ResetIrsVDSCommand, log id: 7e9ccd8a >2012-12-17 08:58:54,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d211c63 >2012-12-17 08:58:54,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:58:54,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d211c63 >2012-12-17 08:59:04,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:04,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4711 >2012-12-17 08:59:04,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4711, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40395881 >2012-12-17 08:59:04,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling started: taskId = bd9f33d7-b55c-4013-b288-9313a0f27bfc >2012-12-17 08:59:05,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling ended: taskId = bd9f33d7-b55c-4013-b288-9313a0f27bfc task status = finished >2012-12-17 08:59:05,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] spmStart polling ended, spm status: SPM >2012-12-17 08:59:05,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [164629b0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bd9f33d7-b55c-4013-b288-9313a0f27bfc), log id: 4dd19dce >2012-12-17 08:59:05,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, HSMClearTaskVDSCommand, log id: 4dd19dce >2012-12-17 08:59:05,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [164629b0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ccf674b, log id: 40395881 >2012-12-17 08:59:06,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:06,103 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:59:06,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 107cc8c0 >2012-12-17 08:59:06,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [164629b0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:59:06,308 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [71b7bdef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:59:06,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 247a2196 >2012-12-17 08:59:06,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3984a49b >2012-12-17 08:59:06,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:06,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStopVDSCommand, log id: 3984a49b >2012-12-17 08:59:06,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, ResetIrsVDSCommand, log id: 247a2196 >2012-12-17 08:59:06,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fb5e0f9 >2012-12-17 08:59:06,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:06,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fb5e0f9 >2012-12-17 08:59:06,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:06,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4712 >2012-12-17 08:59:06,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4712, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28616981 >2012-12-17 08:59:06,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 3b310586-cac4-468a-a124-fe2262d64538 >2012-12-17 08:59:07,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 3b310586-cac4-468a-a124-fe2262d64538 task status = finished >2012-12-17 08:59:07,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM >2012-12-17 08:59:07,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3b310586-cac4-468a-a124-fe2262d64538), log id: 4e1bcc8b >2012-12-17 08:59:07,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 4e1bcc8b >2012-12-17 08:59:07,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@311e1bd, log id: 28616981 >2012-12-17 08:59:07,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:07,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:07,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11d522b7 >2012-12-17 08:59:07,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11d522b7 >2012-12-17 08:59:07,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 107cc8c0 >2012-12-17 08:59:07,915 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:07,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fda2586 >2012-12-17 08:59:07,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:07,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d809042 >2012-12-17 08:59:07,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d809042 >2012-12-17 08:59:07,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fda2586 >2012-12-17 08:59:07,936 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [7582e2ab] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:14,887 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:59:16,803 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3d206f3c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:59:16,945 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [a029edc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:59:17,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 83d0ca7 >2012-12-17 08:59:17,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e2fe42d >2012-12-17 08:59:17,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:17,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStopVDSCommand, log id: 7e2fe42d >2012-12-17 08:59:17,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, ResetIrsVDSCommand, log id: 83d0ca7 >2012-12-17 08:59:17,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ff37726 >2012-12-17 08:59:17,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:17,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ff37726 >2012-12-17 08:59:25,079 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:59:27,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:27,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4713 >2012-12-17 08:59:27,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4713, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3aa6adc2 >2012-12-17 08:59:27,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling started: taskId = 92d6899e-daea-4dee-9e1c-d06c90d33c36 >2012-12-17 08:59:28,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling ended: taskId = 92d6899e-daea-4dee-9e1c-d06c90d33c36 task status = finished >2012-12-17 08:59:28,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] spmStart polling ended, spm status: SPM >2012-12-17 08:59:28,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=92d6899e-daea-4dee-9e1c-d06c90d33c36), log id: 7d8a9dcb >2012-12-17 08:59:28,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, HSMClearTaskVDSCommand, log id: 7d8a9dcb >2012-12-17 08:59:28,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [33e4b3f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@177dbad1, log id: 3aa6adc2 >2012-12-17 08:59:28,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:28,662 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:59:28,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a928470 >2012-12-17 08:59:28,662 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [33e4b3f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:59:28,816 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [39739d98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:59:28,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76d00446 >2012-12-17 08:59:28,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51ced073 >2012-12-17 08:59:28,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:29,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStopVDSCommand, log id: 51ced073 >2012-12-17 08:59:29,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, ResetIrsVDSCommand, log id: 76d00446 >2012-12-17 08:59:29,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4479c005 >2012-12-17 08:59:29,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:29,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4479c005 >2012-12-17 08:59:29,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:29,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4714 >2012-12-17 08:59:29,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4714, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5420893f >2012-12-17 08:59:29,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling started: taskId = ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3 >2012-12-17 08:59:30,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended: taskId = ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3 task status = finished >2012-12-17 08:59:30,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended, spm status: SPM >2012-12-17 08:59:30,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ee4c5cc7-eeeb-4e2a-9b29-d8bc504431a3), log id: 45e3ce5f >2012-12-17 08:59:30,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMClearTaskVDSCommand, log id: 45e3ce5f >2012-12-17 08:59:30,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f01e2d1, log id: 5420893f >2012-12-17 08:59:31,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:31,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:31,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f037e02 >2012-12-17 08:59:31,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f037e02 >2012-12-17 08:59:31,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a928470 >2012-12-17 08:59:31,243 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:31,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 507c5f4f >2012-12-17 08:59:31,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:31,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bcd3ff7 >2012-12-17 08:59:31,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bcd3ff7 >2012-12-17 08:59:31,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 507c5f4f >2012-12-17 08:59:31,263 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [61136831] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:35,316 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:59:39,544 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [3e3e3ebc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:59:39,667 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:59:39,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a8e1387 >2012-12-17 08:59:39,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7719dc70 >2012-12-17 08:59:39,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:39,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, SpmStopVDSCommand, log id: 7719dc70 >2012-12-17 08:59:39,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, ResetIrsVDSCommand, log id: a8e1387 >2012-12-17 08:59:39,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c00db98 >2012-12-17 08:59:39,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:39,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [53d0bc2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c00db98 >2012-12-17 08:59:46,824 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 08:59:49,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:50,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4715 >2012-12-17 08:59:50,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4715, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50e10840 >2012-12-17 08:59:50,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling started: taskId = eec005ab-96df-40ce-95d6-a56f3ab45491 >2012-12-17 08:59:51,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended: taskId = eec005ab-96df-40ce-95d6-a56f3ab45491 task status = finished >2012-12-17 08:59:51,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] spmStart polling ended, spm status: SPM >2012-12-17 08:59:51,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eec005ab-96df-40ce-95d6-a56f3ab45491), log id: 6aeb99cc >2012-12-17 08:59:51,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, HSMClearTaskVDSCommand, log id: 6aeb99cc >2012-12-17 08:59:51,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [28319bba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11be56eb, log id: 50e10840 >2012-12-17 08:59:51,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:51,151 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 08:59:51,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 135619dd >2012-12-17 08:59:51,167 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [28319bba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 08:59:51,251 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [579f6ed1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 08:59:51,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 729aec05 >2012-12-17 08:59:51,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 233aacaf >2012-12-17 08:59:51,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:51,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, SpmStopVDSCommand, log id: 233aacaf >2012-12-17 08:59:51,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, ResetIrsVDSCommand, log id: 729aec05 >2012-12-17 08:59:51,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b5956c0 >2012-12-17 08:59:51,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:51,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [579f6ed1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b5956c0 >2012-12-17 08:59:51,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 08:59:51,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4716 >2012-12-17 08:59:51,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4716, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b3c099c >2012-12-17 08:59:51,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling started: taskId = 1619d4d3-e156-4798-80d0-7a178c49dc16 >2012-12-17 08:59:52,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended: taskId = 1619d4d3-e156-4798-80d0-7a178c49dc16 task status = finished >2012-12-17 08:59:53,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended, spm status: SPM >2012-12-17 08:59:53,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1619d4d3-e156-4798-80d0-7a178c49dc16), log id: f9c7ef8 >2012-12-17 08:59:53,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMClearTaskVDSCommand, log id: f9c7ef8 >2012-12-17 08:59:53,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b53d3c9, log id: 6b3c099c >2012-12-17 08:59:53,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 08:59:53,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:53,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78ff06fa >2012-12-17 08:59:53,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78ff06fa >2012-12-17 08:59:53,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 135619dd >2012-12-17 08:59:53,085 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:53,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f234525 >2012-12-17 08:59:53,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 08:59:53,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 119fc441 >2012-12-17 08:59:53,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 119fc441 >2012-12-17 08:59:53,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f234525 >2012-12-17 08:59:53,105 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-42) [27a8ed97] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 08:59:57,059 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Checking autorecoverable hosts >2012-12-17 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Autorecovering 0 hosts >2012-12-17 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Checking autorecoverable hosts done >2012-12-17 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6bb29080] Autorecovering storage domains is disabled, skipping >2012-12-17 09:00:01,966 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [5a014659] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:02,099 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [7790038] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:02,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4bc1c8fc >2012-12-17 09:00:02,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b0a0f69 >2012-12-17 09:00:02,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:02,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStopVDSCommand, log id: b0a0f69 >2012-12-17 09:00:02,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, ResetIrsVDSCommand, log id: 4bc1c8fc >2012-12-17 09:00:02,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e7f4a72 >2012-12-17 09:00:02,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:02,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e7f4a72 >2012-12-17 09:00:07,267 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:00:12,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:00:12,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4717 >2012-12-17 09:00:12,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4717, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c59ba03 >2012-12-17 09:00:12,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling started: taskId = a7888054-b858-429e-a546-d5fb66202c8f >2012-12-17 09:00:13,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended: taskId = a7888054-b858-429e-a546-d5fb66202c8f task status = finished >2012-12-17 09:00:13,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] spmStart polling ended, spm status: SPM >2012-12-17 09:00:13,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a7888054-b858-429e-a546-d5fb66202c8f), log id: fe4c4c5 >2012-12-17 09:00:13,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, HSMClearTaskVDSCommand, log id: fe4c4c5 >2012-12-17 09:00:13,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [e4c8c37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@743269a9, log id: 7c59ba03 >2012-12-17 09:00:13,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:00:13,577 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:00:13,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 517b692 >2012-12-17 09:00:13,597 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [e4c8c37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:13,688 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [680d007a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:13,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63909f88 >2012-12-17 09:00:13,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23d36a1b >2012-12-17 09:00:13,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:13,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, SpmStopVDSCommand, log id: 23d36a1b >2012-12-17 09:00:13,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, ResetIrsVDSCommand, log id: 63909f88 >2012-12-17 09:00:13,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [680d007a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 241b9e33 >2012-12-17 09:00:13,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:13,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [680d007a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 241b9e33 >2012-12-17 09:00:14,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:00:14,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4718 >2012-12-17 09:00:14,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4718, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2edbbe5e >2012-12-17 09:00:14,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling started: taskId = 0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80 >2012-12-17 09:00:15,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended: taskId = 0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80 task status = finished >2012-12-17 09:00:15,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] spmStart polling ended, spm status: SPM >2012-12-17 09:00:15,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0d6106c8-0b6d-4ff3-be15-cbe6ca1a3b80), log id: 6cdb0cac >2012-12-17 09:00:15,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMClearTaskVDSCommand, log id: 6cdb0cac >2012-12-17 09:00:15,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@211d2809, log id: 2edbbe5e >2012-12-17 09:00:15,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:00:15,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:15,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 851b771 >2012-12-17 09:00:15,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 851b771 >2012-12-17 09:00:15,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 517b692 >2012-12-17 09:00:15,150 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:00:15,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 420b6ec2 >2012-12-17 09:00:15,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:15,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3441c536 >2012-12-17 09:00:15,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3441c536 >2012-12-17 09:00:15,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [5ab536e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 420b6ec2 >2012-12-17 09:00:15,170 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [5ab536e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:00:17,544 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:00:24,050 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [6d961413] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:24,153 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [7781d588] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:24,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6286e2a6 >2012-12-17 09:00:24,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63d935fe >2012-12-17 09:00:24,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:24,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SpmStopVDSCommand, log id: 63d935fe >2012-12-17 09:00:24,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, ResetIrsVDSCommand, log id: 6286e2a6 >2012-12-17 09:00:24,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 994de38 >2012-12-17 09:00:24,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:24,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 994de38 >2012-12-17 09:00:27,752 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:00:34,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:00:34,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4719 >2012-12-17 09:00:34,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4719, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7cde0a5d >2012-12-17 09:00:34,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling started: taskId = a246e9df-90a1-4272-af66-ad6cc984bd70 >2012-12-17 09:00:35,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling ended: taskId = a246e9df-90a1-4272-af66-ad6cc984bd70 task status = finished >2012-12-17 09:00:35,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] spmStart polling ended, spm status: SPM >2012-12-17 09:00:35,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a246e9df-90a1-4272-af66-ad6cc984bd70), log id: 13b3e0dd >2012-12-17 09:00:35,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, HSMClearTaskVDSCommand, log id: 13b3e0dd >2012-12-17 09:00:35,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [1a0f716] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66fd746, log id: 7cde0a5d >2012-12-17 09:00:35,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:00:36,569 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:00:36,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11c49f61 >2012-12-17 09:00:36,714 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1a0f716] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:36,931 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:36,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cf0fb7a >2012-12-17 09:00:36,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 201f3121 >2012-12-17 09:00:37,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:37,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, SpmStopVDSCommand, log id: 201f3121 >2012-12-17 09:00:37,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, ResetIrsVDSCommand, log id: 4cf0fb7a >2012-12-17 09:00:37,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d489bac >2012-12-17 09:00:37,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:37,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d489bac >2012-12-17 09:00:37,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:00:37,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4720 >2012-12-17 09:00:37,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4720, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16895bbb >2012-12-17 09:00:37,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = f9357a86-f6a5-48c1-a89b-bc5896da3391 >2012-12-17 09:00:38,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = f9357a86-f6a5-48c1-a89b-bc5896da3391 task status = finished >2012-12-17 09:00:38,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM >2012-12-17 09:00:38,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9357a86-f6a5-48c1-a89b-bc5896da3391), log id: 26103050 >2012-12-17 09:00:38,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 26103050 >2012-12-17 09:00:38,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70d3e7b5, log id: 16895bbb >2012-12-17 09:00:38,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:00:38,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:38,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: ec27655 >2012-12-17 09:00:38,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: ec27655 >2012-12-17 09:00:38,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11c49f61 >2012-12-17 09:00:38,374 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:00:38,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 30dfaab9 >2012-12-17 09:00:38,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:38,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d2875e5 >2012-12-17 09:00:38,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d2875e5 >2012-12-17 09:00:38,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 30dfaab9 >2012-12-17 09:00:38,394 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:00:47,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [536655ff] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:47,581 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [30604458] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:47,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45d783f1 >2012-12-17 09:00:47,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 19ab309f >2012-12-17 09:00:47,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:48,207 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:00:48,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, SpmStopVDSCommand, log id: 19ab309f >2012-12-17 09:00:48,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, ResetIrsVDSCommand, log id: 45d783f1 >2012-12-17 09:00:48,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 280d1527 >2012-12-17 09:00:48,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:00:48,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 280d1527 >2012-12-17 09:00:58,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:00:58,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4721 >2012-12-17 09:00:58,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4721, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67382da >2012-12-17 09:00:58,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling started: taskId = 9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f >2012-12-17 09:00:59,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling ended: taskId = 9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f task status = finished >2012-12-17 09:00:59,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] spmStart polling ended, spm status: SPM >2012-12-17 09:00:59,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9a64b5b1-0cb4-44bf-a0b3-14ac2a250c2f), log id: 27e66ecd >2012-12-17 09:00:59,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, HSMClearTaskVDSCommand, log id: 27e66ecd >2012-12-17 09:00:59,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [ee8ec0b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9746932, log id: 67382da >2012-12-17 09:00:59,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:00:59,713 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:00:59,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15f85a76 >2012-12-17 09:00:59,735 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [ee8ec0b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:00:59,842 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [50b67ada] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:00:59,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 185f49a6 >2012-12-17 09:00:59,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 16630bec >2012-12-17 09:00:59,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, SpmStopVDSCommand, log id: 16630bec >2012-12-17 09:01:00,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, ResetIrsVDSCommand, log id: 185f49a6 >2012-12-17 09:01:00,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d85f31a >2012-12-17 09:01:00,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:00,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [50b67ada] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d85f31a >2012-12-17 09:01:00,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:01:00,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4722 >2012-12-17 09:01:00,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4722, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6772d648 >2012-12-17 09:01:00,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 19050ad5-45d2-4e6e-9ece-2319ec223c12 >2012-12-17 09:01:01,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 19050ad5-45d2-4e6e-9ece-2319ec223c12 task status = finished >2012-12-17 09:01:01,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM >2012-12-17 09:01:01,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19050ad5-45d2-4e6e-9ece-2319ec223c12), log id: 7be95813 >2012-12-17 09:01:01,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 7be95813 >2012-12-17 09:01:01,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@149586f3, log id: 6772d648 >2012-12-17 09:01:01,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:01:01,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:01,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ba570 >2012-12-17 09:01:01,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ba570 >2012-12-17 09:01:01,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15f85a76 >2012-12-17 09:01:01,443 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:01,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 663d4975 >2012-12-17 09:01:01,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:01,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b8fe719 >2012-12-17 09:01:01,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b8fe719 >2012-12-17 09:01:01,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 663d4975 >2012-12-17 09:01:01,463 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:08,674 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:01:10,363 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3e5a5a03] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:01:11,334 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [4fb47a41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:01:11,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3afb4c29 >2012-12-17 09:01:11,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 650e8b48 >2012-12-17 09:01:11,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:11,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStopVDSCommand, log id: 650e8b48 >2012-12-17 09:01:11,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, ResetIrsVDSCommand, log id: 3afb4c29 >2012-12-17 09:01:11,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ddd8111 >2012-12-17 09:01:11,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:11,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ddd8111 >2012-12-17 09:01:18,900 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:01:21,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:01:21,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4723 >2012-12-17 09:01:21,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4723, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65d0e230 >2012-12-17 09:01:21,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling started: taskId = 909b1e66-2d4b-4293-b99d-4682c47ba929 >2012-12-17 09:01:22,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling ended: taskId = 909b1e66-2d4b-4293-b99d-4682c47ba929 task status = finished >2012-12-17 09:01:22,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] spmStart polling ended, spm status: SPM >2012-12-17 09:01:22,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=909b1e66-2d4b-4293-b99d-4682c47ba929), log id: eaf1bcc >2012-12-17 09:01:22,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, HSMClearTaskVDSCommand, log id: eaf1bcc >2012-12-17 09:01:22,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [67ce2e5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a49f59c, log id: 65d0e230 >2012-12-17 09:01:22,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:01:22,991 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:01:22,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36f818e6 >2012-12-17 09:01:23,021 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [67ce2e5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:01:23,133 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [32d89e5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:01:23,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54379061 >2012-12-17 09:01:23,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2d120fea >2012-12-17 09:01:23,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:24,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SpmStopVDSCommand, log id: 2d120fea >2012-12-17 09:01:24,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, ResetIrsVDSCommand, log id: 54379061 >2012-12-17 09:01:24,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 590012ac >2012-12-17 09:01:24,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:24,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 590012ac >2012-12-17 09:01:24,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:01:24,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4724 >2012-12-17 09:01:24,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4724, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e187dc8 >2012-12-17 09:01:24,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling started: taskId = 5483c80b-689a-4de1-ad4e-e136cdbdb812 >2012-12-17 09:01:25,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended: taskId = 5483c80b-689a-4de1-ad4e-e136cdbdb812 task status = finished >2012-12-17 09:01:25,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended, spm status: SPM >2012-12-17 09:01:25,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5483c80b-689a-4de1-ad4e-e136cdbdb812), log id: 3535176a >2012-12-17 09:01:25,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMClearTaskVDSCommand, log id: 3535176a >2012-12-17 09:01:25,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30944b70, log id: 2e187dc8 >2012-12-17 09:01:25,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:01:25,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:25,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c61f68d >2012-12-17 09:01:25,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c61f68d >2012-12-17 09:01:25,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36f818e6 >2012-12-17 09:01:25,360 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:25,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1381da92 >2012-12-17 09:01:25,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:25,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d44894d >2012-12-17 09:01:25,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d44894d >2012-12-17 09:01:25,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1381da92 >2012-12-17 09:01:25,380 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5bc809fe] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:29,118 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:01:34,224 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [d0709d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:01:34,395 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [25c7cf42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:01:34,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 27a929a8 >2012-12-17 09:01:34,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c40bc14 >2012-12-17 09:01:34,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:34,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SpmStopVDSCommand, log id: 6c40bc14 >2012-12-17 09:01:34,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, ResetIrsVDSCommand, log id: 27a929a8 >2012-12-17 09:01:34,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 763b5eb0 >2012-12-17 09:01:34,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:34,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 763b5eb0 >2012-12-17 09:01:39,366 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:01:44,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:01:44,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4725 >2012-12-17 09:01:44,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4725, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36cbc8f >2012-12-17 09:01:44,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling started: taskId = 95a0afad-bb7b-446c-afe0-51628a88a307 >2012-12-17 09:01:45,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling ended: taskId = 95a0afad-bb7b-446c-afe0-51628a88a307 task status = finished >2012-12-17 09:01:45,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] spmStart polling ended, spm status: SPM >2012-12-17 09:01:45,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95a0afad-bb7b-446c-afe0-51628a88a307), log id: 4336a12f >2012-12-17 09:01:45,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, HSMClearTaskVDSCommand, log id: 4336a12f >2012-12-17 09:01:45,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3ef697d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37903722, log id: 36cbc8f >2012-12-17 09:01:45,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:01:46,190 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:01:46,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11ebf61a >2012-12-17 09:01:46,212 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3ef697d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:01:46,425 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [19e66a45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:01:46,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49a934b9 >2012-12-17 09:01:46,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2b8dafe6 >2012-12-17 09:01:46,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:46,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, SpmStopVDSCommand, log id: 2b8dafe6 >2012-12-17 09:01:46,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, ResetIrsVDSCommand, log id: 49a934b9 >2012-12-17 09:01:46,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 799e162b >2012-12-17 09:01:46,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:46,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 799e162b >2012-12-17 09:01:46,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:01:46,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4726 >2012-12-17 09:01:46,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4726, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fad596e >2012-12-17 09:01:46,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling started: taskId = 597723ef-9024-403b-b691-7daaa3684fa5 >2012-12-17 09:01:47,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended: taskId = 597723ef-9024-403b-b691-7daaa3684fa5 task status = finished >2012-12-17 09:01:47,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] spmStart polling ended, spm status: SPM >2012-12-17 09:01:47,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=597723ef-9024-403b-b691-7daaa3684fa5), log id: b6a5fb3 >2012-12-17 09:01:47,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMClearTaskVDSCommand, log id: b6a5fb3 >2012-12-17 09:01:47,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74b0e69f, log id: 4fad596e >2012-12-17 09:01:47,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:01:47,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:47,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e586a35 >2012-12-17 09:01:47,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e586a35 >2012-12-17 09:01:47,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 11ebf61a >2012-12-17 09:01:47,923 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:47,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ea4dd8d >2012-12-17 09:01:47,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:47,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 797adcb9 >2012-12-17 09:01:47,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 797adcb9 >2012-12-17 09:01:47,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [2324f948] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ea4dd8d >2012-12-17 09:01:47,942 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [2324f948] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:01:49,606 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:01:56,794 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6cb72173] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:01:56,882 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [1c12d57c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:01:56,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22f9919a >2012-12-17 09:01:56,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 223b5ba4 >2012-12-17 09:01:56,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:57,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, SpmStopVDSCommand, log id: 223b5ba4 >2012-12-17 09:01:57,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, ResetIrsVDSCommand, log id: 22f9919a >2012-12-17 09:01:57,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ae0b5bc >2012-12-17 09:01:57,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1c12d57c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:01:57,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1c12d57c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ae0b5bc >2012-12-17 09:01:59,799 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:02:07,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:07,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4727 >2012-12-17 09:02:07,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4727, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45954737 >2012-12-17 09:02:07,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling started: taskId = c67ed1ea-524f-42cf-854a-65380aca6edc >2012-12-17 09:02:08,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling ended: taskId = c67ed1ea-524f-42cf-854a-65380aca6edc task status = finished >2012-12-17 09:02:08,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] spmStart polling ended, spm status: SPM >2012-12-17 09:02:08,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c67ed1ea-524f-42cf-854a-65380aca6edc), log id: 1c2c3f69 >2012-12-17 09:02:08,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, HSMClearTaskVDSCommand, log id: 1c2c3f69 >2012-12-17 09:02:08,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [32c2c5a9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1cf714f2, log id: 45954737 >2012-12-17 09:02:08,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:08,378 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:02:08,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c8ac91f >2012-12-17 09:02:08,395 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [32c2c5a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:02:08,583 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:02:08,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44cf968a >2012-12-17 09:02:08,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 36ac3816 >2012-12-17 09:02:08,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:09,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, SpmStopVDSCommand, log id: 36ac3816 >2012-12-17 09:02:09,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, ResetIrsVDSCommand, log id: 44cf968a >2012-12-17 09:02:09,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 292269af >2012-12-17 09:02:09,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:09,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 292269af >2012-12-17 09:02:09,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4728 >2012-12-17 09:02:09,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4728, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47e37e3c >2012-12-17 09:02:09,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling started: taskId = 17456770-127a-450e-9c90-76948fd1c7ee >2012-12-17 09:02:10,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling ended: taskId = 17456770-127a-450e-9c90-76948fd1c7ee task status = finished >2012-12-17 09:02:10,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] spmStart polling ended, spm status: SPM >2012-12-17 09:02:10,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=17456770-127a-450e-9c90-76948fd1c7ee), log id: 6722c6d6 >2012-12-17 09:02:10,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMClearTaskVDSCommand, log id: 6722c6d6 >2012-12-17 09:02:10,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c2c4080, log id: 47e37e3c >2012-12-17 09:02:10,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:10,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:10,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22f32b19 >2012-12-17 09:02:10,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22f32b19 >2012-12-17 09:02:10,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c8ac91f >2012-12-17 09:02:10,947 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:02:10,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a92883c >2012-12-17 09:02:10,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:10,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e36259d >2012-12-17 09:02:10,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e36259d >2012-12-17 09:02:10,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [55ed84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a92883c >2012-12-17 09:02:10,967 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [55ed84] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:02:19,306 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [49b46a39] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:02:19,396 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [505260a3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:02:19,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 416a8c3b >2012-12-17 09:02:19,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 225c9b93 >2012-12-17 09:02:19,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:19,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SpmStopVDSCommand, log id: 225c9b93 >2012-12-17 09:02:19,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, ResetIrsVDSCommand, log id: 416a8c3b >2012-12-17 09:02:19,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c1478a4 >2012-12-17 09:02:19,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:19,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c1478a4 >2012-12-17 09:02:20,220 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:02:29,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:29,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4729 >2012-12-17 09:02:29,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4729, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b332b95 >2012-12-17 09:02:29,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling started: taskId = 2e5233b6-3088-46a7-bcf1-4222a037bb4a >2012-12-17 09:02:30,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling ended: taskId = 2e5233b6-3088-46a7-bcf1-4222a037bb4a task status = finished >2012-12-17 09:02:30,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] spmStart polling ended, spm status: SPM >2012-12-17 09:02:30,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e5233b6-3088-46a7-bcf1-4222a037bb4a), log id: 56cecef1 >2012-12-17 09:02:30,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, HSMClearTaskVDSCommand, log id: 56cecef1 >2012-12-17 09:02:30,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [67d5cb38] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3672c99e, log id: 6b332b95 >2012-12-17 09:02:31,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:31,568 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:02:31,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5073e296 >2012-12-17 09:02:31,600 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [67d5cb38] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:02:31,756 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [57f1ef14] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:02:31,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 55c490a6 >2012-12-17 09:02:31,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 369f524 >2012-12-17 09:02:31,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:31,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, SpmStopVDSCommand, log id: 369f524 >2012-12-17 09:02:31,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, ResetIrsVDSCommand, log id: 55c490a6 >2012-12-17 09:02:32,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f3e82ca >2012-12-17 09:02:32,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [57f1ef14] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:32,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [57f1ef14] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f3e82ca >2012-12-17 09:02:32,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:32,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4730 >2012-12-17 09:02:32,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4730, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5bb87508 >2012-12-17 09:02:32,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling started: taskId = 58a3130d-3f95-45c5-8a02-5d8440985fd2 >2012-12-17 09:02:33,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling ended: taskId = 58a3130d-3f95-45c5-8a02-5d8440985fd2 task status = finished >2012-12-17 09:02:33,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] spmStart polling ended, spm status: SPM >2012-12-17 09:02:33,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=58a3130d-3f95-45c5-8a02-5d8440985fd2), log id: 648a41ea >2012-12-17 09:02:33,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMClearTaskVDSCommand, log id: 648a41ea >2012-12-17 09:02:33,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30ed9d37, log id: 5bb87508 >2012-12-17 09:02:33,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:33,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:33,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a78df43 >2012-12-17 09:02:33,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a78df43 >2012-12-17 09:02:33,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5073e296 >2012-12-17 09:02:33,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:02:33,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 481254fd >2012-12-17 09:02:33,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:33,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2467b64b >2012-12-17 09:02:33,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2467b64b >2012-12-17 09:02:33,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 481254fd >2012-12-17 09:02:33,227 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) [2ea5eaf2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:02:41,688 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:02:42,113 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6fcdea84] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:02:42,238 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [6bebc102] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:02:42,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2348513c >2012-12-17 09:02:42,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24ce4457 >2012-12-17 09:02:42,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:42,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, SpmStopVDSCommand, log id: 24ce4457 >2012-12-17 09:02:42,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, ResetIrsVDSCommand, log id: 2348513c >2012-12-17 09:02:42,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a3ce1e0 >2012-12-17 09:02:42,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:42,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a3ce1e0 >2012-12-17 09:02:51,866 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:02:52,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:52,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4731 >2012-12-17 09:02:52,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4731, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6fcf8237 >2012-12-17 09:02:52,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling started: taskId = cb044a7b-50a1-4283-9c1d-9fc80c045f9b >2012-12-17 09:02:53,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended: taskId = cb044a7b-50a1-4283-9c1d-9fc80c045f9b task status = finished >2012-12-17 09:02:53,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] spmStart polling ended, spm status: SPM >2012-12-17 09:02:53,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb044a7b-50a1-4283-9c1d-9fc80c045f9b), log id: 2a55a455 >2012-12-17 09:02:53,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, HSMClearTaskVDSCommand, log id: 2a55a455 >2012-12-17 09:02:53,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [155e23b1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c97de29, log id: 6fcf8237 >2012-12-17 09:02:53,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:53,742 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:02:53,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 686a43e2 >2012-12-17 09:02:53,771 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [155e23b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:02:53,887 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [757df811] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:02:53,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 421aa903 >2012-12-17 09:02:53,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 314bf78d >2012-12-17 09:02:53,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:54,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, SpmStopVDSCommand, log id: 314bf78d >2012-12-17 09:02:54,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, ResetIrsVDSCommand, log id: 421aa903 >2012-12-17 09:02:54,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [757df811] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f67378d >2012-12-17 09:02:54,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [757df811] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:54,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [757df811] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f67378d >2012-12-17 09:02:54,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:02:54,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4732 >2012-12-17 09:02:54,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4732, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b837fc7 >2012-12-17 09:02:54,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling started: taskId = f28b0566-733f-4f12-9fb8-12340561123a >2012-12-17 09:02:55,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended: taskId = f28b0566-733f-4f12-9fb8-12340561123a task status = finished >2012-12-17 09:02:55,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended, spm status: SPM >2012-12-17 09:02:55,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f28b0566-733f-4f12-9fb8-12340561123a), log id: 604639a >2012-12-17 09:02:55,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMClearTaskVDSCommand, log id: 604639a >2012-12-17 09:02:55,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a27168c, log id: 2b837fc7 >2012-12-17 09:02:55,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:02:55,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:55,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 220231cd >2012-12-17 09:02:55,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 220231cd >2012-12-17 09:02:55,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 686a43e2 >2012-12-17 09:02:55,505 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:02:55,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e139b2c >2012-12-17 09:02:55,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:02:55,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32144f57 >2012-12-17 09:02:55,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32144f57 >2012-12-17 09:02:55,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e139b2c >2012-12-17 09:02:55,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [441eec2e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:03:02,083 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:04,273 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [786fe4b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:03:04,420 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [72736c3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:03:04,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17c8d79a >2012-12-17 09:03:04,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 752c98b >2012-12-17 09:03:04,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:04,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, SpmStopVDSCommand, log id: 752c98b >2012-12-17 09:03:04,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, ResetIrsVDSCommand, log id: 17c8d79a >2012-12-17 09:03:04,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66c9aeaf >2012-12-17 09:03:04,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [72736c3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:04,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [72736c3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66c9aeaf >2012-12-17 09:03:12,282 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:14,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:03:14,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4733 >2012-12-17 09:03:14,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4733, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7627d942 >2012-12-17 09:03:14,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling started: taskId = 102587cf-f5f1-4973-98c6-81f0be820d82 >2012-12-17 09:03:15,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling ended: taskId = 102587cf-f5f1-4973-98c6-81f0be820d82 task status = finished >2012-12-17 09:03:15,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] spmStart polling ended, spm status: SPM >2012-12-17 09:03:15,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [15510694] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=102587cf-f5f1-4973-98c6-81f0be820d82), log id: 1d8e2ef9 >2012-12-17 09:03:15,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, HSMClearTaskVDSCommand, log id: 1d8e2ef9 >2012-12-17 09:03:15,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [15510694] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61adec97, log id: 7627d942 >2012-12-17 09:03:16,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:03:16,145 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:03:16,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 341244a9 >2012-12-17 09:03:16,180 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [15510694] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:03:16,328 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [3808661d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:03:16,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 32b7c0eb >2012-12-17 09:03:16,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e73bf18 >2012-12-17 09:03:16,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:16,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, SpmStopVDSCommand, log id: 5e73bf18 >2012-12-17 09:03:16,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, ResetIrsVDSCommand, log id: 32b7c0eb >2012-12-17 09:03:16,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3808661d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b2c98da >2012-12-17 09:03:16,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:16,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3808661d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b2c98da >2012-12-17 09:03:16,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:03:16,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4734 >2012-12-17 09:03:16,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4734, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4eac4dc6 >2012-12-17 09:03:16,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling started: taskId = 2fd5d38f-ba56-4eec-804b-9d9a31821346 >2012-12-17 09:03:17,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended: taskId = 2fd5d38f-ba56-4eec-804b-9d9a31821346 task status = finished >2012-12-17 09:03:17,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended, spm status: SPM >2012-12-17 09:03:17,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2fd5d38f-ba56-4eec-804b-9d9a31821346), log id: 22005947 >2012-12-17 09:03:17,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMClearTaskVDSCommand, log id: 22005947 >2012-12-17 09:03:17,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@44e555c1, log id: 4eac4dc6 >2012-12-17 09:03:17,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:03:17,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:17,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21ebd76d >2012-12-17 09:03:17,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21ebd76d >2012-12-17 09:03:17,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 341244a9 >2012-12-17 09:03:17,745 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:03:17,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4768c075 >2012-12-17 09:03:17,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:17,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c1aba78 >2012-12-17 09:03:17,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c1aba78 >2012-12-17 09:03:17,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4768c075 >2012-12-17 09:03:17,765 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [5e38fcf2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:03:22,474 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:26,677 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [107aac36] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:03:26,787 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:03:26,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2cdc1d0e >2012-12-17 09:03:26,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3cdb2c49 >2012-12-17 09:03:26,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:27,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, SpmStopVDSCommand, log id: 3cdb2c49 >2012-12-17 09:03:27,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, ResetIrsVDSCommand, log id: 2cdc1d0e >2012-12-17 09:03:27,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13364e48 >2012-12-17 09:03:27,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:27,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7e0f01b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13364e48 >2012-12-17 09:03:32,683 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:37,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:03:37,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4735 >2012-12-17 09:03:37,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4735, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c1b5910 >2012-12-17 09:03:37,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling started: taskId = e9132787-165c-4a55-afb7-a64717521835 >2012-12-17 09:03:38,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling ended: taskId = e9132787-165c-4a55-afb7-a64717521835 task status = finished >2012-12-17 09:03:38,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] spmStart polling ended, spm status: SPM >2012-12-17 09:03:38,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [490300dd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9132787-165c-4a55-afb7-a64717521835), log id: 78dee0f7 >2012-12-17 09:03:38,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, HSMClearTaskVDSCommand, log id: 78dee0f7 >2012-12-17 09:03:38,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [490300dd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7083b61f, log id: 3c1b5910 >2012-12-17 09:03:38,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:03:38,676 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:03:38,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ba6a883 >2012-12-17 09:03:38,718 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [490300dd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:03:38,886 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [2c8e27f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:03:38,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3cf273ed >2012-12-17 09:03:38,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4508b00d >2012-12-17 09:03:38,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:39,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, SpmStopVDSCommand, log id: 4508b00d >2012-12-17 09:03:39,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, ResetIrsVDSCommand, log id: 3cf273ed >2012-12-17 09:03:39,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 676a54d >2012-12-17 09:03:39,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2c8e27f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:39,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2c8e27f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 676a54d >2012-12-17 09:03:39,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:03:39,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4736 >2012-12-17 09:03:39,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4736, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f6e157a >2012-12-17 09:03:39,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 57270bb2-8d30-4b32-9feb-eb3f4537d5dd >2012-12-17 09:03:40,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 57270bb2-8d30-4b32-9feb-eb3f4537d5dd task status = finished >2012-12-17 09:03:40,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 09:03:40,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=57270bb2-8d30-4b32-9feb-eb3f4537d5dd), log id: 5ba082a2 >2012-12-17 09:03:40,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 5ba082a2 >2012-12-17 09:03:40,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e088ad, log id: 6f6e157a >2012-12-17 09:03:40,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:03:41,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:41,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d12b777 >2012-12-17 09:03:41,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d12b777 >2012-12-17 09:03:41,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ba6a883 >2012-12-17 09:03:41,311 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:03:41,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27cafcbf >2012-12-17 09:03:41,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:41,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b00a409 >2012-12-17 09:03:41,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b00a409 >2012-12-17 09:03:41,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27cafcbf >2012-12-17 09:03:41,331 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:03:42,881 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:49,031 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-14) [3d41845a] Running command: LogoutUserCommand internal: false. >2012-12-17 09:03:49,259 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [579f6ed1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:03:49,354 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [94e4f4d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:03:49,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 369b804d >2012-12-17 09:03:49,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e80bed9 >2012-12-17 09:03:49,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:49,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, SpmStopVDSCommand, log id: e80bed9 >2012-12-17 09:03:49,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, ResetIrsVDSCommand, log id: 369b804d >2012-12-17 09:03:49,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ae17413 >2012-12-17 09:03:49,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:03:49,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [94e4f4d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ae17413 >2012-12-17 09:03:53,072 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:03:59,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:03:59,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4737 >2012-12-17 09:03:59,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4737, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 49bd1acf >2012-12-17 09:03:59,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling started: taskId = 25bc733a-33b3-4a04-ba77-d5fc3cf4d525 >2012-12-17 09:04:00,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling ended: taskId = 25bc733a-33b3-4a04-ba77-d5fc3cf4d525 task status = finished >2012-12-17 09:04:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] spmStart polling ended, spm status: SPM >2012-12-17 09:04:00,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [30604458] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25bc733a-33b3-4a04-ba77-d5fc3cf4d525), log id: 268a5401 >2012-12-17 09:04:00,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, HSMClearTaskVDSCommand, log id: 268a5401 >2012-12-17 09:04:00,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [30604458] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4391216b, log id: 49bd1acf >2012-12-17 09:04:01,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:01,204 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:04:01,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 663c46a0 >2012-12-17 09:04:01,223 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [30604458] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:01,386 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [33cde09c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:01,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 450cf449 >2012-12-17 09:04:01,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ef98b19 >2012-12-17 09:04:01,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:01,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStopVDSCommand, log id: 3ef98b19 >2012-12-17 09:04:01,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, ResetIrsVDSCommand, log id: 450cf449 >2012-12-17 09:04:01,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3591cab8 >2012-12-17 09:04:01,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:01,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3591cab8 >2012-12-17 09:04:01,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:04:01,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4738 >2012-12-17 09:04:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4738, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 489a2fa5 >2012-12-17 09:04:01,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling started: taskId = 06e27fcc-d6ac-4c25-848e-d3b754444ae5 >2012-12-17 09:04:02,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended: taskId = 06e27fcc-d6ac-4c25-848e-d3b754444ae5 task status = finished >2012-12-17 09:04:02,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] spmStart polling ended, spm status: SPM >2012-12-17 09:04:02,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06e27fcc-d6ac-4c25-848e-d3b754444ae5), log id: 2c6dae83 >2012-12-17 09:04:02,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMClearTaskVDSCommand, log id: 2c6dae83 >2012-12-17 09:04:02,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54dc313e, log id: 489a2fa5 >2012-12-17 09:04:02,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:02,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:02,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14392b54 >2012-12-17 09:04:02,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14392b54 >2012-12-17 09:04:02,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 663c46a0 >2012-12-17 09:04:02,821 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:02,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7883d062 >2012-12-17 09:04:02,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b1d8a50 >2012-12-17 09:04:02,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b1d8a50 >2012-12-17 09:04:02,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [7f520532] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7883d062 >2012-12-17 09:04:02,841 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [7f520532] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:03,278 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:04:11,723 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [2324f948] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:11,822 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [13d04fcc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:11,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b9e4925 >2012-12-17 09:04:11,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38610a45 >2012-12-17 09:04:11,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:12,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, SpmStopVDSCommand, log id: 38610a45 >2012-12-17 09:04:12,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, ResetIrsVDSCommand, log id: 6b9e4925 >2012-12-17 09:04:12,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5687f090 >2012-12-17 09:04:12,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:12,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5687f090 >2012-12-17 09:04:13,503 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:04:22,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:04:22,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4739 >2012-12-17 09:04:22,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4739, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 156bb43f >2012-12-17 09:04:22,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling started: taskId = 8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75 >2012-12-17 09:04:23,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended: taskId = 8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75 task status = finished >2012-12-17 09:04:23,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] spmStart polling ended, spm status: SPM >2012-12-17 09:04:23,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a9fe74e-4acf-4bdb-87c2-dfc9984e5f75), log id: 4bc6e580 >2012-12-17 09:04:23,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, HSMClearTaskVDSCommand, log id: 4bc6e580 >2012-12-17 09:04:23,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [46701658] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f055d5d, log id: 156bb43f >2012-12-17 09:04:23,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:23,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:04:23,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1355fada >2012-12-17 09:04:23,392 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [46701658] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:23,507 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [261e40a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:23,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7381198a >2012-12-17 09:04:23,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7721e4df >2012-12-17 09:04:23,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:23,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStopVDSCommand, log id: 7721e4df >2012-12-17 09:04:23,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, ResetIrsVDSCommand, log id: 7381198a >2012-12-17 09:04:23,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26424ab7 >2012-12-17 09:04:23,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:23,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26424ab7 >2012-12-17 09:04:23,790 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:04:23,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:04:23,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4740 >2012-12-17 09:04:23,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4740, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c40e8fb >2012-12-17 09:04:23,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling started: taskId = 368cb029-b962-47c4-80ab-22fc9d728d50 >2012-12-17 09:04:24,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended: taskId = 368cb029-b962-47c4-80ab-22fc9d728d50 task status = finished >2012-12-17 09:04:24,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended, spm status: SPM >2012-12-17 09:04:24,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=368cb029-b962-47c4-80ab-22fc9d728d50), log id: a66634 >2012-12-17 09:04:24,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMClearTaskVDSCommand, log id: a66634 >2012-12-17 09:04:24,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c7fe041, log id: c40e8fb >2012-12-17 09:04:24,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:25,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:25,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: cfadf1c >2012-12-17 09:04:25,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: cfadf1c >2012-12-17 09:04:25,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1355fada >2012-12-17 09:04:25,030 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:25,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a4dadec >2012-12-17 09:04:25,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:25,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f961a5e >2012-12-17 09:04:25,034 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:04:25,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2d03bf49 >2012-12-17 09:04:25,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f961a5e >2012-12-17 09:04:25,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a4dadec >2012-12-17 09:04:25,050 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [4fb47a41] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:25,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:25,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d62d5fd >2012-12-17 09:04:25,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d62d5fd >2012-12-17 09:04:25,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2d03bf49 >2012-12-17 09:04:25,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:25,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cf04f1 >2012-12-17 09:04:25,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:25,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e6c9393 >2012-12-17 09:04:25,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e6c9393 >2012-12-17 09:04:25,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cf04f1 >2012-12-17 09:04:25,091 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [46a80193] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:33,942 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55ed84] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:34,222 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [670d1744] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:34,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dc6613c >2012-12-17 09:04:34,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c3530fd >2012-12-17 09:04:34,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:34,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, SpmStopVDSCommand, log id: c3530fd >2012-12-17 09:04:34,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, ResetIrsVDSCommand, log id: dc6613c >2012-12-17 09:04:34,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b8cc53a >2012-12-17 09:04:34,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:34,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b8cc53a >2012-12-17 09:04:44,287 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:04:44,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:04:44,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4741 >2012-12-17 09:04:44,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4741, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6da82b42 >2012-12-17 09:04:44,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling started: taskId = 3cc41031-fe0b-438f-99bf-f31af2183103 >2012-12-17 09:04:45,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling ended: taskId = 3cc41031-fe0b-438f-99bf-f31af2183103 task status = finished >2012-12-17 09:04:45,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] spmStart polling ended, spm status: SPM >2012-12-17 09:04:45,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3cc41031-fe0b-438f-99bf-f31af2183103), log id: 94fc038 >2012-12-17 09:04:45,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, HSMClearTaskVDSCommand, log id: 94fc038 >2012-12-17 09:04:45,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5abe3acd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52d4a14e, log id: 6da82b42 >2012-12-17 09:04:46,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:46,460 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:04:46,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74540a08 >2012-12-17 09:04:46,505 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5abe3acd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:46,684 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [371b8996] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:46,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c79be22 >2012-12-17 09:04:46,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3312ca9e >2012-12-17 09:04:46,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:47,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, SpmStopVDSCommand, log id: 3312ca9e >2012-12-17 09:04:47,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, ResetIrsVDSCommand, log id: 7c79be22 >2012-12-17 09:04:47,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [371b8996] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d8a840f >2012-12-17 09:04:47,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [371b8996] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:47,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [371b8996] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d8a840f >2012-12-17 09:04:47,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:04:47,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4742 >2012-12-17 09:04:47,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4742, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13e5687a >2012-12-17 09:04:47,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = c060be6f-7791-4aae-b5d9-8ec4157f7f52 >2012-12-17 09:04:48,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = c060be6f-7791-4aae-b5d9-8ec4157f7f52 task status = finished >2012-12-17 09:04:48,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM >2012-12-17 09:04:48,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c060be6f-7791-4aae-b5d9-8ec4157f7f52), log id: 598b4d83 >2012-12-17 09:04:48,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 598b4d83 >2012-12-17 09:04:48,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@488e0162, log id: 13e5687a >2012-12-17 09:04:48,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:04:48,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:48,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 611b0ea8 >2012-12-17 09:04:48,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 611b0ea8 >2012-12-17 09:04:48,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74540a08 >2012-12-17 09:04:48,811 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:48,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5900c3f0 >2012-12-17 09:04:48,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:48,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a906286 >2012-12-17 09:04:48,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a906286 >2012-12-17 09:04:48,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5900c3f0 >2012-12-17 09:04:48,831 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:04:54,534 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:04:57,650 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [50b67ada] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:04:58,012 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [57cdd675] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:04:58,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7ef78ee >2012-12-17 09:04:58,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7f61f52d >2012-12-17 09:04:58,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:58,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStopVDSCommand, log id: 7f61f52d >2012-12-17 09:04:58,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, ResetIrsVDSCommand, log id: 7ef78ee >2012-12-17 09:04:58,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6802d1a9 >2012-12-17 09:04:58,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:04:58,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6802d1a9 >2012-12-17 09:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Checking autorecoverable hosts >2012-12-17 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Autorecovering 0 hosts >2012-12-17 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Checking autorecoverable hosts done >2012-12-17 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [37630ba1] Autorecovering storage domains is disabled, skipping >2012-12-17 09:05:04,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:08,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:08,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4743 >2012-12-17 09:05:08,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4743, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 428e22e9 >2012-12-17 09:05:08,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling started: taskId = 6b114bce-dc6d-4c5e-ba29-107e9efedff0 >2012-12-17 09:05:09,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling ended: taskId = 6b114bce-dc6d-4c5e-ba29-107e9efedff0 task status = finished >2012-12-17 09:05:09,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] spmStart polling ended, spm status: SPM >2012-12-17 09:05:09,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [108329de] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b114bce-dc6d-4c5e-ba29-107e9efedff0), log id: 6ddcee7b >2012-12-17 09:05:09,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, HSMClearTaskVDSCommand, log id: 6ddcee7b >2012-12-17 09:05:09,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [108329de] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43ad0596, log id: 428e22e9 >2012-12-17 09:05:09,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:09,599 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:05:09,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6532d26c >2012-12-17 09:05:09,618 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [108329de] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:05:09,735 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [73cd7795] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:05:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 41f6c478 >2012-12-17 09:05:09,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7559310a >2012-12-17 09:05:09,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:09,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, SpmStopVDSCommand, log id: 7559310a >2012-12-17 09:05:09,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, ResetIrsVDSCommand, log id: 41f6c478 >2012-12-17 09:05:09,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 205ca197 >2012-12-17 09:05:09,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:09,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 205ca197 >2012-12-17 09:05:10,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:10,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4744 >2012-12-17 09:05:10,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4744, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7dac2afd >2012-12-17 09:05:10,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling started: taskId = 9dbd1ce7-6978-4cfd-82a4-aabf9c32447d >2012-12-17 09:05:11,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended: taskId = 9dbd1ce7-6978-4cfd-82a4-aabf9c32447d task status = finished >2012-12-17 09:05:11,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended, spm status: SPM >2012-12-17 09:05:11,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9dbd1ce7-6978-4cfd-82a4-aabf9c32447d), log id: 74fd0b4d >2012-12-17 09:05:11,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMClearTaskVDSCommand, log id: 74fd0b4d >2012-12-17 09:05:11,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10f1283c, log id: 7dac2afd >2012-12-17 09:05:11,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:11,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:11,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71261222 >2012-12-17 09:05:11,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71261222 >2012-12-17 09:05:11,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6532d26c >2012-12-17 09:05:11,225 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:11,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a70b036 >2012-12-17 09:05:11,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:11,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 422adb4b >2012-12-17 09:05:11,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 422adb4b >2012-12-17 09:05:11,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7a70b036 >2012-12-17 09:05:11,246 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [261e40a1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:14,932 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:20,082 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [48c4b2a7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:05:20,297 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [3b88f2b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:05:20,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5733c68b >2012-12-17 09:05:20,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25df7980 >2012-12-17 09:05:20,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:20,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, SpmStopVDSCommand, log id: 25df7980 >2012-12-17 09:05:21,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, ResetIrsVDSCommand, log id: 5733c68b >2012-12-17 09:05:21,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d601072 >2012-12-17 09:05:21,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [3b88f2b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:21,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [3b88f2b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d601072 >2012-12-17 09:05:25,169 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:31,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:31,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4745 >2012-12-17 09:05:31,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4745, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 679c6a22 >2012-12-17 09:05:31,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling started: taskId = 274cc9cf-51b7-4394-bfd6-d07379fdbfa7 >2012-12-17 09:05:32,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended: taskId = 274cc9cf-51b7-4394-bfd6-d07379fdbfa7 task status = finished >2012-12-17 09:05:32,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] spmStart polling ended, spm status: SPM >2012-12-17 09:05:32,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=274cc9cf-51b7-4394-bfd6-d07379fdbfa7), log id: 79ef136c >2012-12-17 09:05:32,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, HSMClearTaskVDSCommand, log id: 79ef136c >2012-12-17 09:05:32,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [cdacbd1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ee455fe, log id: 679c6a22 >2012-12-17 09:05:32,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:32,424 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:05:32,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e93a643 >2012-12-17 09:05:32,440 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [cdacbd1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:05:32,623 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [537845d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:05:32,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 377cc50a >2012-12-17 09:05:32,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ba6b79a >2012-12-17 09:05:32,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:33,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, SpmStopVDSCommand, log id: 5ba6b79a >2012-12-17 09:05:33,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, ResetIrsVDSCommand, log id: 377cc50a >2012-12-17 09:05:33,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6693a3f2 >2012-12-17 09:05:33,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:33,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6693a3f2 >2012-12-17 09:05:33,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:33,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4746 >2012-12-17 09:05:33,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4746, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2530890b >2012-12-17 09:05:33,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = 1e301a73-da6e-4d79-aa12-4767f7a52065 >2012-12-17 09:05:34,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = 1e301a73-da6e-4d79-aa12-4767f7a52065 task status = finished >2012-12-17 09:05:34,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM >2012-12-17 09:05:34,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e301a73-da6e-4d79-aa12-4767f7a52065), log id: 129d346b >2012-12-17 09:05:34,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 129d346b >2012-12-17 09:05:34,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@edc6659, log id: 2530890b >2012-12-17 09:05:34,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:34,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:34,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 50f6fd6d >2012-12-17 09:05:34,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50f6fd6d >2012-12-17 09:05:34,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e93a643 >2012-12-17 09:05:34,468 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:34,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77828a2 >2012-12-17 09:05:34,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:34,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55e861d9 >2012-12-17 09:05:34,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e861d9 >2012-12-17 09:05:34,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77828a2 >2012-12-17 09:05:34,488 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:35,379 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:43,352 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [15ef2937] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:05:43,533 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [1d6be49e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:05:43,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 58e597d6 >2012-12-17 09:05:43,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 125d71c >2012-12-17 09:05:43,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:43,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SpmStopVDSCommand, log id: 125d71c >2012-12-17 09:05:43,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, ResetIrsVDSCommand, log id: 58e597d6 >2012-12-17 09:05:43,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27e22339 >2012-12-17 09:05:43,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:43,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27e22339 >2012-12-17 09:05:46,195 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:53,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:53,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4747 >2012-12-17 09:05:53,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4747, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23f0c438 >2012-12-17 09:05:53,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling started: taskId = e6a86dc0-f041-41c0-8cbe-b334f098e16c >2012-12-17 09:05:54,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling ended: taskId = e6a86dc0-f041-41c0-8cbe-b334f098e16c task status = finished >2012-12-17 09:05:55,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] spmStart polling ended, spm status: SPM >2012-12-17 09:05:55,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e6a86dc0-f041-41c0-8cbe-b334f098e16c), log id: a4151a9 >2012-12-17 09:05:55,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, HSMClearTaskVDSCommand, log id: a4151a9 >2012-12-17 09:05:55,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [6bebc102] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46583381, log id: 23f0c438 >2012-12-17 09:05:55,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:55,079 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:05:55,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69841251 >2012-12-17 09:05:55,101 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [6bebc102] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:05:55,188 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:05:55,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e48ddae >2012-12-17 09:05:55,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ed3459 >2012-12-17 09:05:55,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:55,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStopVDSCommand, log id: 7ed3459 >2012-12-17 09:05:56,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, ResetIrsVDSCommand, log id: 6e48ddae >2012-12-17 09:05:56,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 517c2211 >2012-12-17 09:05:56,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:56,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 517c2211 >2012-12-17 09:05:56,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:05:56,428 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:05:56,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4748 >2012-12-17 09:05:56,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4748, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3cd6b18a >2012-12-17 09:05:56,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling started: taskId = 1998cb60-f0b3-480a-86b7-8368342962ee >2012-12-17 09:05:57,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended: taskId = 1998cb60-f0b3-480a-86b7-8368342962ee task status = finished >2012-12-17 09:05:57,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended, spm status: SPM >2012-12-17 09:05:57,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1998cb60-f0b3-480a-86b7-8368342962ee), log id: 18ab78ee >2012-12-17 09:05:57,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMClearTaskVDSCommand, log id: 18ab78ee >2012-12-17 09:05:57,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2debfa98, log id: 3cd6b18a >2012-12-17 09:05:57,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:05:57,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:57,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5ab1db3b >2012-12-17 09:05:57,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5ab1db3b >2012-12-17 09:05:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69841251 >2012-12-17 09:05:57,617 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52b02d01 >2012-12-17 09:05:57,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:57,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c16fa37 >2012-12-17 09:05:57,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c16fa37 >2012-12-17 09:05:57,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52b02d01 >2012-12-17 09:05:57,637 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [1925ecb1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:57,660 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:05:57,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: c6903e0 >2012-12-17 09:05:57,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:57,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 394f69fb >2012-12-17 09:05:57,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 394f69fb >2012-12-17 09:05:57,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c6903e0 >2012-12-17 09:05:57,680 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:05:57,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: cb6fb22 >2012-12-17 09:05:57,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:05:57,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5e63dbce >2012-12-17 09:05:57,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5e63dbce >2012-12-17 09:05:57,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [7781d588] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: cb6fb22 >2012-12-17 09:05:57,700 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [7781d588] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:06:06,517 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [33e6033] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:06:06,860 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:06:07,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a10484a >2012-12-17 09:06:07,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 17bbe054 >2012-12-17 09:06:07,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:07,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, SpmStopVDSCommand, log id: 17bbe054 >2012-12-17 09:06:07,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, ResetIrsVDSCommand, log id: 5a10484a >2012-12-17 09:06:07,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e22bae >2012-12-17 09:06:07,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:07,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e22bae >2012-12-17 09:06:16,916 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:06:17,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:06:17,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4749 >2012-12-17 09:06:17,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4749, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f940983 >2012-12-17 09:06:17,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling started: taskId = 2e94a711-bbc2-4413-a701-b62972a1755b >2012-12-17 09:06:18,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended: taskId = 2e94a711-bbc2-4413-a701-b62972a1755b task status = finished >2012-12-17 09:06:18,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] spmStart polling ended, spm status: SPM >2012-12-17 09:06:18,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2e94a711-bbc2-4413-a701-b62972a1755b), log id: 3ffc83ea >2012-12-17 09:06:18,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, HSMClearTaskVDSCommand, log id: 3ffc83ea >2012-12-17 09:06:18,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [7582e2ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b389877, log id: 7f940983 >2012-12-17 09:06:18,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:06:18,529 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:06:18,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e72cfdf >2012-12-17 09:06:18,548 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [7582e2ab] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:06:18,791 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [1b349a52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:06:18,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39630de2 >2012-12-17 09:06:18,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14d4425f >2012-12-17 09:06:18,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:19,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, SpmStopVDSCommand, log id: 14d4425f >2012-12-17 09:06:19,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, ResetIrsVDSCommand, log id: 39630de2 >2012-12-17 09:06:19,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cad9b84 >2012-12-17 09:06:19,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:19,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cad9b84 >2012-12-17 09:06:19,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:06:19,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4750 >2012-12-17 09:06:19,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4750, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 639aa8b8 >2012-12-17 09:06:19,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling started: taskId = 6dc0c6ec-0cd3-410d-b049-9f99fc365a49 >2012-12-17 09:06:20,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended: taskId = 6dc0c6ec-0cd3-410d-b049-9f99fc365a49 task status = finished >2012-12-17 09:06:20,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended, spm status: SPM >2012-12-17 09:06:20,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6dc0c6ec-0cd3-410d-b049-9f99fc365a49), log id: 260db526 >2012-12-17 09:06:20,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMClearTaskVDSCommand, log id: 260db526 >2012-12-17 09:06:20,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4defe54e, log id: 639aa8b8 >2012-12-17 09:06:20,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:06:20,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:20,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10563007 >2012-12-17 09:06:20,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10563007 >2012-12-17 09:06:20,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e72cfdf >2012-12-17 09:06:20,352 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:06:20,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a6b23f2 >2012-12-17 09:06:20,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:20,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 324427c1 >2012-12-17 09:06:20,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 324427c1 >2012-12-17 09:06:20,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a6b23f2 >2012-12-17 09:06:20,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [b3b23b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:06:27,169 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:06:29,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [57f1ef14] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:06:29,353 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [2e31d140] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:06:29,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21d27e46 >2012-12-17 09:06:29,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51061dfc >2012-12-17 09:06:29,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:29,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStopVDSCommand, log id: 51061dfc >2012-12-17 09:06:29,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, ResetIrsVDSCommand, log id: 21d27e46 >2012-12-17 09:06:29,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bb04690 >2012-12-17 09:06:29,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:29,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bb04690 >2012-12-17 09:06:37,435 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:06:39,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:06:39,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4751 >2012-12-17 09:06:39,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4751, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fad926e >2012-12-17 09:06:39,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling started: taskId = 537abddb-57ca-4efb-9163-4f70498ce443 >2012-12-17 09:06:40,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling ended: taskId = 537abddb-57ca-4efb-9163-4f70498ce443 task status = finished >2012-12-17 09:06:40,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] spmStart polling ended, spm status: SPM >2012-12-17 09:06:40,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=537abddb-57ca-4efb-9163-4f70498ce443), log id: 662e6f05 >2012-12-17 09:06:40,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, HSMClearTaskVDSCommand, log id: 662e6f05 >2012-12-17 09:06:40,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [41dfde4b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@727ae5ab, log id: 2fad926e >2012-12-17 09:06:41,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:06:41,706 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:06:41,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a32c779 >2012-12-17 09:06:41,721 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [41dfde4b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:06:41,916 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [1d6e3277] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:06:41,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 645d055f >2012-12-17 09:06:41,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3972d14c >2012-12-17 09:06:41,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:42,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, SpmStopVDSCommand, log id: 3972d14c >2012-12-17 09:06:42,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, ResetIrsVDSCommand, log id: 645d055f >2012-12-17 09:06:42,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 120ba2a7 >2012-12-17 09:06:42,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:42,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [1d6e3277] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 120ba2a7 >2012-12-17 09:06:42,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:06:42,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4752 >2012-12-17 09:06:42,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4752, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44efeb30 >2012-12-17 09:06:42,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling started: taskId = 4b9dc5ec-8793-4a53-b631-19bb1e2e25f4 >2012-12-17 09:06:43,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling ended: taskId = 4b9dc5ec-8793-4a53-b631-19bb1e2e25f4 task status = finished >2012-12-17 09:06:43,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] spmStart polling ended, spm status: SPM >2012-12-17 09:06:43,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b9dc5ec-8793-4a53-b631-19bb1e2e25f4), log id: 1345c758 >2012-12-17 09:06:43,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMClearTaskVDSCommand, log id: 1345c758 >2012-12-17 09:06:43,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4a226c4a, log id: 44efeb30 >2012-12-17 09:06:43,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:06:43,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:43,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48d5d5e6 >2012-12-17 09:06:43,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48d5d5e6 >2012-12-17 09:06:43,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a32c779 >2012-12-17 09:06:43,414 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:06:43,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ece1465 >2012-12-17 09:06:43,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:43,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 14df502b >2012-12-17 09:06:43,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14df502b >2012-12-17 09:06:43,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-34) [5dcd105e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ece1465 >2012-12-17 09:06:43,438 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-34) [5dcd105e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:06:47,629 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:06:52,283 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [419474a0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:06:52,385 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [5d11909d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:06:52,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 62f0bdbf >2012-12-17 09:06:52,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39ccd91f >2012-12-17 09:06:52,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:52,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, SpmStopVDSCommand, log id: 39ccd91f >2012-12-17 09:06:52,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, ResetIrsVDSCommand, log id: 62f0bdbf >2012-12-17 09:06:52,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75de7371 >2012-12-17 09:06:52,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:06:52,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75de7371 >2012-12-17 09:06:57,827 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:07:02,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:02,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4753 >2012-12-17 09:07:02,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4753, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bd00e2d >2012-12-17 09:07:02,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling started: taskId = 70b61ba2-bda1-490d-8228-74eb5da0d29c >2012-12-17 09:07:03,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended: taskId = 70b61ba2-bda1-490d-8228-74eb5da0d29c task status = finished >2012-12-17 09:07:03,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] spmStart polling ended, spm status: SPM >2012-12-17 09:07:03,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70b61ba2-bda1-490d-8228-74eb5da0d29c), log id: 44ba4750 >2012-12-17 09:07:03,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, HSMClearTaskVDSCommand, log id: 44ba4750 >2012-12-17 09:07:03,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [27a8ed97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219ff870, log id: 3bd00e2d >2012-12-17 09:07:03,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:03,916 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:07:03,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5eba94e1 >2012-12-17 09:07:03,956 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [27a8ed97] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:07:04,069 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [455c512f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:07:04,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37f9d7a >2012-12-17 09:07:04,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 716d05b8 >2012-12-17 09:07:04,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:04,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, SpmStopVDSCommand, log id: 716d05b8 >2012-12-17 09:07:04,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, ResetIrsVDSCommand, log id: 37f9d7a >2012-12-17 09:07:04,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [455c512f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f787bd9 >2012-12-17 09:07:04,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [455c512f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:04,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [455c512f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f787bd9 >2012-12-17 09:07:04,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:04,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4754 >2012-12-17 09:07:04,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4754, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28bd51cd >2012-12-17 09:07:04,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling started: taskId = ec9ad9d5-ed19-4b93-bfd4-b31691786cde >2012-12-17 09:07:05,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling ended: taskId = ec9ad9d5-ed19-4b93-bfd4-b31691786cde task status = finished >2012-12-17 09:07:05,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] spmStart polling ended, spm status: SPM >2012-12-17 09:07:05,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ec9ad9d5-ed19-4b93-bfd4-b31691786cde), log id: 48ff5700 >2012-12-17 09:07:05,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMClearTaskVDSCommand, log id: 48ff5700 >2012-12-17 09:07:05,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73a8e670, log id: 28bd51cd >2012-12-17 09:07:06,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:06,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:06,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52ef296 >2012-12-17 09:07:06,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52ef296 >2012-12-17 09:07:06,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5eba94e1 >2012-12-17 09:07:06,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:06,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4d385882 >2012-12-17 09:07:06,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:06,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32cec836 >2012-12-17 09:07:06,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32cec836 >2012-12-17 09:07:06,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [505260a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4d385882 >2012-12-17 09:07:06,470 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [505260a3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:08,138 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:07:14,551 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3808661d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:07:14,701 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [3aea96b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:07:14,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52bdee72 >2012-12-17 09:07:14,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 239578d >2012-12-17 09:07:14,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:14,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStopVDSCommand, log id: 239578d >2012-12-17 09:07:14,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, ResetIrsVDSCommand, log id: 52bdee72 >2012-12-17 09:07:14,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76abf691 >2012-12-17 09:07:14,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:14,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76abf691 >2012-12-17 09:07:18,397 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:07:25,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:25,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4755 >2012-12-17 09:07:25,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4755, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a75936 >2012-12-17 09:07:25,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling started: taskId = 423a7f09-5bd2-4134-98bd-17669110ec38 >2012-12-17 09:07:26,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended: taskId = 423a7f09-5bd2-4134-98bd-17669110ec38 task status = finished >2012-12-17 09:07:26,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] spmStart polling ended, spm status: SPM >2012-12-17 09:07:26,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=423a7f09-5bd2-4134-98bd-17669110ec38), log id: 43153e71 >2012-12-17 09:07:26,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, HSMClearTaskVDSCommand, log id: 43153e71 >2012-12-17 09:07:26,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [181d86f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27b68f1f, log id: 5a75936 >2012-12-17 09:07:26,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:26,268 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:07:26,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 618f5025 >2012-12-17 09:07:26,280 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [181d86f1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:07:26,528 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [af8a779] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:07:26,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 769de3fe >2012-12-17 09:07:26,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 596d4e74 >2012-12-17 09:07:26,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:27,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, SpmStopVDSCommand, log id: 596d4e74 >2012-12-17 09:07:27,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, ResetIrsVDSCommand, log id: 769de3fe >2012-12-17 09:07:27,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [af8a779] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c1316f8 >2012-12-17 09:07:27,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [af8a779] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:27,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [af8a779] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c1316f8 >2012-12-17 09:07:27,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:27,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4756 >2012-12-17 09:07:27,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4756, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d62b2b7 >2012-12-17 09:07:27,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling started: taskId = 18dfe81b-0b38-45d0-9463-2705024911f6 >2012-12-17 09:07:28,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended: taskId = 18dfe81b-0b38-45d0-9463-2705024911f6 task status = finished >2012-12-17 09:07:28,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended, spm status: SPM >2012-12-17 09:07:28,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=18dfe81b-0b38-45d0-9463-2705024911f6), log id: 7f957c50 >2012-12-17 09:07:28,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMClearTaskVDSCommand, log id: 7f957c50 >2012-12-17 09:07:28,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f14a9a0, log id: 7d62b2b7 >2012-12-17 09:07:28,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:28,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:28,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65e87494 >2012-12-17 09:07:28,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65e87494 >2012-12-17 09:07:28,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 618f5025 >2012-12-17 09:07:28,362 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:28,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dd4d1 >2012-12-17 09:07:28,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:28,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 307bc725 >2012-12-17 09:07:28,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 307bc725 >2012-12-17 09:07:28,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dd4d1 >2012-12-17 09:07:28,383 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:28,609 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:07:37,194 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2c8e27f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:07:37,300 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [6573753] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:07:37,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5004452c >2012-12-17 09:07:37,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26caba79 >2012-12-17 09:07:37,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:38,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, SpmStopVDSCommand, log id: 26caba79 >2012-12-17 09:07:38,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, ResetIrsVDSCommand, log id: 5004452c >2012-12-17 09:07:38,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5053d0 >2012-12-17 09:07:38,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:38,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5053d0 >2012-12-17 09:07:38,865 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:07:48,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:48,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4757 >2012-12-17 09:07:48,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4757, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ca4e3d3 >2012-12-17 09:07:48,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling started: taskId = cca469a2-f66e-4170-bdc1-7ef45e6634f4 >2012-12-17 09:07:49,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended: taskId = cca469a2-f66e-4170-bdc1-7ef45e6634f4 task status = finished >2012-12-17 09:07:49,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] spmStart polling ended, spm status: SPM >2012-12-17 09:07:49,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cca469a2-f66e-4170-bdc1-7ef45e6634f4), log id: 55386c30 >2012-12-17 09:07:49,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, HSMClearTaskVDSCommand, log id: 55386c30 >2012-12-17 09:07:49,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5bc809fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b46ef05, log id: 5ca4e3d3 >2012-12-17 09:07:49,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:49,440 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:07:49,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 338767a6 >2012-12-17 09:07:49,480 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5bc809fe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:07:49,690 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [263a433b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:07:49,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1164ba76 >2012-12-17 09:07:49,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 280d1f6d >2012-12-17 09:07:49,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:49,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStopVDSCommand, log id: 280d1f6d >2012-12-17 09:07:49,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, ResetIrsVDSCommand, log id: 1164ba76 >2012-12-17 09:07:49,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71a5d43b >2012-12-17 09:07:49,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:49,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71a5d43b >2012-12-17 09:07:49,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:07:49,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4758 >2012-12-17 09:07:50,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4758, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53c1bc43 >2012-12-17 09:07:50,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling started: taskId = efa98d38-80d1-4999-821b-ac7da9b7ed51 >2012-12-17 09:07:51,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended: taskId = efa98d38-80d1-4999-821b-ac7da9b7ed51 task status = finished >2012-12-17 09:07:51,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended, spm status: SPM >2012-12-17 09:07:51,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=efa98d38-80d1-4999-821b-ac7da9b7ed51), log id: 63c3e81d >2012-12-17 09:07:51,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMClearTaskVDSCommand, log id: 63c3e81d >2012-12-17 09:07:51,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52bbfb16, log id: 53c1bc43 >2012-12-17 09:07:51,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:07:51,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:51,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ca38fa7 >2012-12-17 09:07:51,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ca38fa7 >2012-12-17 09:07:51,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 338767a6 >2012-12-17 09:07:51,232 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:51,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 321a018c >2012-12-17 09:07:51,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:07:51,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2de2cc91 >2012-12-17 09:07:51,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2de2cc91 >2012-12-17 09:07:51,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 321a018c >2012-12-17 09:07:51,252 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [71b7bdef] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:07:59,257 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:00,035 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [7f520532] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:00,255 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [24cba1b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:00,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4197239 >2012-12-17 09:08:00,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49618fb7 >2012-12-17 09:08:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:00,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, SpmStopVDSCommand, log id: 49618fb7 >2012-12-17 09:08:00,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, ResetIrsVDSCommand, log id: 4197239 >2012-12-17 09:08:00,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ea29a16 >2012-12-17 09:08:00,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:00,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ea29a16 >2012-12-17 09:08:09,538 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:11,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:11,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4759 >2012-12-17 09:08:11,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4759, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e7f11d >2012-12-17 09:08:11,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling started: taskId = 6791b8d5-ee11-41cb-97ce-b1e603db1259 >2012-12-17 09:08:12,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling ended: taskId = 6791b8d5-ee11-41cb-97ce-b1e603db1259 task status = finished >2012-12-17 09:08:12,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] spmStart polling ended, spm status: SPM >2012-12-17 09:08:12,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6791b8d5-ee11-41cb-97ce-b1e603db1259), log id: cab48dc >2012-12-17 09:08:12,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, HSMClearTaskVDSCommand, log id: cab48dc >2012-12-17 09:08:12,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [13d04fcc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@413c6a03, log id: e7f11d >2012-12-17 09:08:12,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:08:12,558 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:08:12,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59462d2d >2012-12-17 09:08:12,585 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [13d04fcc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:12,699 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [30572eed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:12,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 31bc4167 >2012-12-17 09:08:12,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 212a69c >2012-12-17 09:08:12,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:12,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, SpmStopVDSCommand, log id: 212a69c >2012-12-17 09:08:12,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, ResetIrsVDSCommand, log id: 31bc4167 >2012-12-17 09:08:12,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4033f314 >2012-12-17 09:08:12,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:12,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4033f314 >2012-12-17 09:08:12,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:12,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4760 >2012-12-17 09:08:12,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4760, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67c97c39 >2012-12-17 09:08:13,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling started: taskId = 64c320dd-1224-4d2d-8244-1d965f1f7cf7 >2012-12-17 09:08:14,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling ended: taskId = 64c320dd-1224-4d2d-8244-1d965f1f7cf7 task status = finished >2012-12-17 09:08:14,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] spmStart polling ended, spm status: SPM >2012-12-17 09:08:14,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=64c320dd-1224-4d2d-8244-1d965f1f7cf7), log id: 2e4d1123 >2012-12-17 09:08:14,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMClearTaskVDSCommand, log id: 2e4d1123 >2012-12-17 09:08:14,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45a02894, log id: 67c97c39 >2012-12-17 09:08:14,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:08:14,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:14,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 341601ea >2012-12-17 09:08:14,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 341601ea >2012-12-17 09:08:14,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59462d2d >2012-12-17 09:08:14,152 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:08:14,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 459cbe4e >2012-12-17 09:08:14,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:14,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 69951ce5 >2012-12-17 09:08:14,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69951ce5 >2012-12-17 09:08:14,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [5044c757] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 459cbe4e >2012-12-17 09:08:14,176 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [5044c757] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:08:19,791 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:23,029 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [53d0bc2b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:23,163 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [68b3db6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:23,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54367191 >2012-12-17 09:08:23,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3f149e9d >2012-12-17 09:08:23,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:23,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SpmStopVDSCommand, log id: 3f149e9d >2012-12-17 09:08:23,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, ResetIrsVDSCommand, log id: 54367191 >2012-12-17 09:08:23,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5889c3b6 >2012-12-17 09:08:23,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:23,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5889c3b6 >2012-12-17 09:08:30,046 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:33,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:33,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4761 >2012-12-17 09:08:33,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4761, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c51721b >2012-12-17 09:08:33,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling started: taskId = b63f19ee-1920-4163-9bbe-155e59ccfc6c >2012-12-17 09:08:34,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling ended: taskId = b63f19ee-1920-4163-9bbe-155e59ccfc6c task status = finished >2012-12-17 09:08:34,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] spmStart polling ended, spm status: SPM >2012-12-17 09:08:34,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [670d1744] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b63f19ee-1920-4163-9bbe-155e59ccfc6c), log id: 4472dd81 >2012-12-17 09:08:34,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, HSMClearTaskVDSCommand, log id: 4472dd81 >2012-12-17 09:08:34,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [670d1744] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@512c6342, log id: 3c51721b >2012-12-17 09:08:34,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:08:34,757 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:08:34,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 524ffe4e >2012-12-17 09:08:34,789 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [670d1744] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:34,970 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [3299bae4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:35,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4636d8b4 >2012-12-17 09:08:35,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62fa569c >2012-12-17 09:08:35,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:35,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStopVDSCommand, log id: 62fa569c >2012-12-17 09:08:35,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, ResetIrsVDSCommand, log id: 4636d8b4 >2012-12-17 09:08:35,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 282f4ab3 >2012-12-17 09:08:35,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:35,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 282f4ab3 >2012-12-17 09:08:35,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:35,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4762 >2012-12-17 09:08:35,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4762, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 43322053 >2012-12-17 09:08:35,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling started: taskId = cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff >2012-12-17 09:08:36,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended: taskId = cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff task status = finished >2012-12-17 09:08:36,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] spmStart polling ended, spm status: SPM >2012-12-17 09:08:36,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cd6172b7-46c9-4ebd-b26f-62fbf7cbcfff), log id: 638add10 >2012-12-17 09:08:36,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMClearTaskVDSCommand, log id: 638add10 >2012-12-17 09:08:36,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31b7787e, log id: 43322053 >2012-12-17 09:08:36,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:08:36,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:36,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25d1978e >2012-12-17 09:08:36,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25d1978e >2012-12-17 09:08:36,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 524ffe4e >2012-12-17 09:08:36,627 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:08:36,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6334ff07 >2012-12-17 09:08:36,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:36,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76761b4e >2012-12-17 09:08:36,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76761b4e >2012-12-17 09:08:36,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [733172b2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6334ff07 >2012-12-17 09:08:36,647 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [733172b2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:08:40,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:46,099 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [371b8996] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:46,780 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [480a798f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:46,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ea1af92 >2012-12-17 09:08:46,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40f63719 >2012-12-17 09:08:46,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:47,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, SpmStopVDSCommand, log id: 40f63719 >2012-12-17 09:08:47,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, ResetIrsVDSCommand, log id: 3ea1af92 >2012-12-17 09:08:47,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 431ea8e8 >2012-12-17 09:08:47,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:47,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 431ea8e8 >2012-12-17 09:08:51,646 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:08:57,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:57,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4763 >2012-12-17 09:08:57,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4763, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5972443f >2012-12-17 09:08:57,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling started: taskId = a71533a1-221e-4d67-9c7e-bcb26c6574d7 >2012-12-17 09:08:58,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling ended: taskId = a71533a1-221e-4d67-9c7e-bcb26c6574d7 task status = finished >2012-12-17 09:08:58,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] spmStart polling ended, spm status: SPM >2012-12-17 09:08:58,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a71533a1-221e-4d67-9c7e-bcb26c6574d7), log id: 79fc1eb3 >2012-12-17 09:08:58,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, HSMClearTaskVDSCommand, log id: 79fc1eb3 >2012-12-17 09:08:58,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [46e0c818] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1743b856, log id: 5972443f >2012-12-17 09:08:58,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:08:58,332 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:08:58,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1203fae8 >2012-12-17 09:08:58,338 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [46e0c818] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:08:58,519 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [779086b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:08:58,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4427ee24 >2012-12-17 09:08:58,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53c11695 >2012-12-17 09:08:58,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:58,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStopVDSCommand, log id: 53c11695 >2012-12-17 09:08:58,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, ResetIrsVDSCommand, log id: 4427ee24 >2012-12-17 09:08:58,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e35fd7f >2012-12-17 09:08:58,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:08:58,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e35fd7f >2012-12-17 09:08:59,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:08:59,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4764 >2012-12-17 09:08:59,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4764, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2070b779 >2012-12-17 09:08:59,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling started: taskId = eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9 >2012-12-17 09:09:00,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended: taskId = eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9 task status = finished >2012-12-17 09:09:00,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended, spm status: SPM >2012-12-17 09:09:00,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb12ad9a-f6de-4f76-b3e0-24eafc4a06d9), log id: 41b3cbe5 >2012-12-17 09:09:00,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMClearTaskVDSCommand, log id: 41b3cbe5 >2012-12-17 09:09:00,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30f4916, log id: 2070b779 >2012-12-17 09:09:00,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:09:00,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6de2b919 >2012-12-17 09:09:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6de2b919 >2012-12-17 09:09:00,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1203fae8 >2012-12-17 09:09:00,194 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:00,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fe650ab >2012-12-17 09:09:00,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:00,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32a07141 >2012-12-17 09:09:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a07141 >2012-12-17 09:09:00,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fe650ab >2012-12-17 09:09:00,215 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [1d8c274e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:01,944 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:09:09,085 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1c12d57c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:09:09,388 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:09:09,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fdddfe3 >2012-12-17 09:09:09,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7acd073a >2012-12-17 09:09:09,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:09,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, SpmStopVDSCommand, log id: 7acd073a >2012-12-17 09:09:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, ResetIrsVDSCommand, log id: 4fdddfe3 >2012-12-17 09:09:09,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10ade151 >2012-12-17 09:09:09,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:09,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10ade151 >2012-12-17 09:09:12,155 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:09:19,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:09:19,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4765 >2012-12-17 09:09:19,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4765, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32802a5f >2012-12-17 09:09:19,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling started: taskId = f1fd9523-c772-41bb-8f64-49054fdbaab0 >2012-12-17 09:09:20,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling ended: taskId = f1fd9523-c772-41bb-8f64-49054fdbaab0 task status = finished >2012-12-17 09:09:20,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] spmStart polling ended, spm status: SPM >2012-12-17 09:09:20,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1fd9523-c772-41bb-8f64-49054fdbaab0), log id: 64371594 >2012-12-17 09:09:20,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, HSMClearTaskVDSCommand, log id: 64371594 >2012-12-17 09:09:20,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c69daf2, log id: 32802a5f >2012-12-17 09:09:21,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:09:21,357 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:09:21,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63722f99 >2012-12-17 09:09:21,478 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3e3bfaaa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:09:21,610 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1000cccb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:09:21,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36a4c590 >2012-12-17 09:09:21,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e4df0d6 >2012-12-17 09:09:21,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:21,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, SpmStopVDSCommand, log id: 6e4df0d6 >2012-12-17 09:09:21,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, ResetIrsVDSCommand, log id: 36a4c590 >2012-12-17 09:09:21,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c6fff72 >2012-12-17 09:09:21,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:21,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1000cccb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c6fff72 >2012-12-17 09:09:21,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:09:21,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4766 >2012-12-17 09:09:21,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4766, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48bcacfa >2012-12-17 09:09:21,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling started: taskId = 2cd466cd-27f4-41b1-a847-e269f21b6f44 >2012-12-17 09:09:22,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling ended: taskId = 2cd466cd-27f4-41b1-a847-e269f21b6f44 task status = finished >2012-12-17 09:09:22,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] spmStart polling ended, spm status: SPM >2012-12-17 09:09:22,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2cd466cd-27f4-41b1-a847-e269f21b6f44), log id: 4bc20056 >2012-12-17 09:09:23,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMClearTaskVDSCommand, log id: 4bc20056 >2012-12-17 09:09:23,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2da434f5, log id: 48bcacfa >2012-12-17 09:09:23,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:09:23,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:23,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78b88ed2 >2012-12-17 09:09:23,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78b88ed2 >2012-12-17 09:09:23,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63722f99 >2012-12-17 09:09:23,078 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:23,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 98725b2 >2012-12-17 09:09:23,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:23,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63f119ed >2012-12-17 09:09:23,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63f119ed >2012-12-17 09:09:23,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [32d89e5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 98725b2 >2012-12-17 09:09:23,098 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [32d89e5e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:31,991 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4256c72b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:09:32,109 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [746c8fa6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:09:32,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 659d8152 >2012-12-17 09:09:32,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1958c16b >2012-12-17 09:09:32,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:32,594 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:09:32,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SpmStopVDSCommand, log id: 1958c16b >2012-12-17 09:09:32,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, ResetIrsVDSCommand, log id: 659d8152 >2012-12-17 09:09:32,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bc66976 >2012-12-17 09:09:32,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:32,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bc66976 >2012-12-17 09:09:42,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:09:42,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4767 >2012-12-17 09:09:42,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4767, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6778ce3f >2012-12-17 09:09:42,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling started: taskId = 5f352215-62a1-42e4-8ad1-793d515ac750 >2012-12-17 09:09:43,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling ended: taskId = 5f352215-62a1-42e4-8ad1-793d515ac750 task status = finished >2012-12-17 09:09:43,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] spmStart polling ended, spm status: SPM >2012-12-17 09:09:43,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f352215-62a1-42e4-8ad1-793d515ac750), log id: f50d1a1 >2012-12-17 09:09:43,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, HSMClearTaskVDSCommand, log id: f50d1a1 >2012-12-17 09:09:43,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [19e66a45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f2a8e2, log id: 6778ce3f >2012-12-17 09:09:43,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:09:43,982 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:09:43,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 265f4369 >2012-12-17 09:09:44,013 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [19e66a45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:09:44,120 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:09:44,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 27b3d4e6 >2012-12-17 09:09:44,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 55dfa68c >2012-12-17 09:09:44,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:44,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, SpmStopVDSCommand, log id: 55dfa68c >2012-12-17 09:09:44,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, ResetIrsVDSCommand, log id: 27b3d4e6 >2012-12-17 09:09:44,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2249bae >2012-12-17 09:09:44,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:44,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2249bae >2012-12-17 09:09:44,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:09:44,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4768 >2012-12-17 09:09:44,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4768, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ff17052 >2012-12-17 09:09:44,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 03d1f9f5-48d7-4385-9ede-e707e7b71f1d >2012-12-17 09:09:45,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 03d1f9f5-48d7-4385-9ede-e707e7b71f1d task status = finished >2012-12-17 09:09:45,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM >2012-12-17 09:09:45,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=03d1f9f5-48d7-4385-9ede-e707e7b71f1d), log id: 72e9658e >2012-12-17 09:09:45,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 72e9658e >2012-12-17 09:09:45,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@390a8892, log id: 4ff17052 >2012-12-17 09:09:45,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:09:45,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:45,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72257f37 >2012-12-17 09:09:45,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72257f37 >2012-12-17 09:09:45,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 265f4369 >2012-12-17 09:09:45,998 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:45,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7bc32e6c >2012-12-17 09:09:45,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:46,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 67a58f37 >2012-12-17 09:09:46,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 67a58f37 >2012-12-17 09:09:46,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7bc32e6c >2012-12-17 09:09:46,018 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [75b7a65b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:09:53,008 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:09:54,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [5dcd105e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:09:54,625 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [7d47f958] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:09:54,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14ff3c7d >2012-12-17 09:09:54,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5c70a8bd >2012-12-17 09:09:54,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:55,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, SpmStopVDSCommand, log id: 5c70a8bd >2012-12-17 09:09:55,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, ResetIrsVDSCommand, log id: 14ff3c7d >2012-12-17 09:09:55,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78465a5e >2012-12-17 09:09:55,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:09:55,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78465a5e >2012-12-17 09:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Checking autorecoverable hosts >2012-12-17 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Autorecovering 0 hosts >2012-12-17 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Checking autorecoverable hosts done >2012-12-17 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7d79337c] Autorecovering storage domains is disabled, skipping >2012-12-17 09:10:03,199 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:10:06,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:06,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4769 >2012-12-17 09:10:06,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4769, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6dde86c2 >2012-12-17 09:10:06,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling started: taskId = 59672a3a-268e-4d7c-af79-8ed3a72614f3 >2012-12-17 09:10:07,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling ended: taskId = 59672a3a-268e-4d7c-af79-8ed3a72614f3 task status = finished >2012-12-17 09:10:07,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] spmStart polling ended, spm status: SPM >2012-12-17 09:10:07,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=59672a3a-268e-4d7c-af79-8ed3a72614f3), log id: 2c297817 >2012-12-17 09:10:07,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, HSMClearTaskVDSCommand, log id: 2c297817 >2012-12-17 09:10:07,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [29a7f0f3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@568b44a7, log id: 6dde86c2 >2012-12-17 09:10:07,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:07,272 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:10:07,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d13e969 >2012-12-17 09:10:07,302 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [29a7f0f3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:10:07,429 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [75aa641b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:10:07,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61a6c9f7 >2012-12-17 09:10:07,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2775d5e0 >2012-12-17 09:10:07,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:07,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, SpmStopVDSCommand, log id: 2775d5e0 >2012-12-17 09:10:07,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, ResetIrsVDSCommand, log id: 61a6c9f7 >2012-12-17 09:10:07,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 618f161 >2012-12-17 09:10:07,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:07,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [75aa641b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 618f161 >2012-12-17 09:10:07,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:07,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4770 >2012-12-17 09:10:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4770, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30c6d720 >2012-12-17 09:10:07,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling started: taskId = 076ccd64-1470-4c6d-9db3-21d3b57dd35e >2012-12-17 09:10:08,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling ended: taskId = 076ccd64-1470-4c6d-9db3-21d3b57dd35e task status = finished >2012-12-17 09:10:08,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] spmStart polling ended, spm status: SPM >2012-12-17 09:10:08,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=076ccd64-1470-4c6d-9db3-21d3b57dd35e), log id: 7c3e312d >2012-12-17 09:10:08,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMClearTaskVDSCommand, log id: 7c3e312d >2012-12-17 09:10:08,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@570ec114, log id: 30c6d720 >2012-12-17 09:10:08,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:08,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:08,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ddbceba >2012-12-17 09:10:08,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ddbceba >2012-12-17 09:10:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d13e969 >2012-12-17 09:10:08,962 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:10:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36b31905 >2012-12-17 09:10:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:08,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d51e944 >2012-12-17 09:10:08,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d51e944 >2012-12-17 09:10:08,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [436410e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36b31905 >2012-12-17 09:10:08,983 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [436410e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:10:13,372 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:10:17,829 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [505260a3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:10:17,981 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [3ff8505] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:10:18,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7dd42974 >2012-12-17 09:10:18,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d5250b5 >2012-12-17 09:10:18,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:18,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStopVDSCommand, log id: 3d5250b5 >2012-12-17 09:10:18,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, ResetIrsVDSCommand, log id: 7dd42974 >2012-12-17 09:10:18,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f6e55e >2012-12-17 09:10:18,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:18,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f6e55e >2012-12-17 09:10:23,606 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:10:28,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:28,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4771 >2012-12-17 09:10:28,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4771, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45b8a6f5 >2012-12-17 09:10:28,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling started: taskId = 47fdb2f6-6497-4e9a-b8f7-8b5591f10873 >2012-12-17 09:10:29,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended: taskId = 47fdb2f6-6497-4e9a-b8f7-8b5591f10873 task status = finished >2012-12-17 09:10:29,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] spmStart polling ended, spm status: SPM >2012-12-17 09:10:29,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=47fdb2f6-6497-4e9a-b8f7-8b5591f10873), log id: 41fc4fb3 >2012-12-17 09:10:29,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, HSMClearTaskVDSCommand, log id: 41fc4fb3 >2012-12-17 09:10:29,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [5e38fcf2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bd4929d, log id: 45b8a6f5 >2012-12-17 09:10:29,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:29,475 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:10:29,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e96bce6 >2012-12-17 09:10:29,545 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5e38fcf2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:10:29,666 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [492f1d2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:10:29,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4076de53 >2012-12-17 09:10:29,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6e3fd33b >2012-12-17 09:10:29,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:30,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, SpmStopVDSCommand, log id: 6e3fd33b >2012-12-17 09:10:30,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, ResetIrsVDSCommand, log id: 4076de53 >2012-12-17 09:10:30,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41c64596 >2012-12-17 09:10:30,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [492f1d2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:30,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [492f1d2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41c64596 >2012-12-17 09:10:30,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:30,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4772 >2012-12-17 09:10:30,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4772, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4af1ac7d >2012-12-17 09:10:30,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = 49e076d4-c32b-4887-a4f4-29c59d54d6be >2012-12-17 09:10:31,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = 49e076d4-c32b-4887-a4f4-29c59d54d6be task status = finished >2012-12-17 09:10:31,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM >2012-12-17 09:10:31,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=49e076d4-c32b-4887-a4f4-29c59d54d6be), log id: 4478235e >2012-12-17 09:10:31,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 4478235e >2012-12-17 09:10:31,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b5a1370, log id: 4af1ac7d >2012-12-17 09:10:31,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:31,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:31,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fc815e0 >2012-12-17 09:10:31,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fc815e0 >2012-12-17 09:10:31,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e96bce6 >2012-12-17 09:10:31,337 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:10:31,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ba4c464 >2012-12-17 09:10:31,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:31,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d614025 >2012-12-17 09:10:31,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d614025 >2012-12-17 09:10:31,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ba4c464 >2012-12-17 09:10:31,358 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:10:33,844 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:10:40,158 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1d6e3277] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:10:41,366 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:10:41,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4529cfaa >2012-12-17 09:10:41,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 28a1da6a >2012-12-17 09:10:41,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:41,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, SpmStopVDSCommand, log id: 28a1da6a >2012-12-17 09:10:41,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, ResetIrsVDSCommand, log id: 4529cfaa >2012-12-17 09:10:41,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2412caca >2012-12-17 09:10:41,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:41,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [14ba2ea4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2412caca >2012-12-17 09:10:44,039 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:10:52,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:52,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4773 >2012-12-17 09:10:52,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4773, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40099959 >2012-12-17 09:10:52,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling started: taskId = 88a6b190-2003-4b24-b4a0-25be1b5ba19f >2012-12-17 09:10:53,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling ended: taskId = 88a6b190-2003-4b24-b4a0-25be1b5ba19f task status = finished >2012-12-17 09:10:53,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] spmStart polling ended, spm status: SPM >2012-12-17 09:10:53,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=88a6b190-2003-4b24-b4a0-25be1b5ba19f), log id: 130279a6 >2012-12-17 09:10:53,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, HSMClearTaskVDSCommand, log id: 130279a6 >2012-12-17 09:10:53,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [5d11909d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7808b0a, log id: 40099959 >2012-12-17 09:10:53,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:53,231 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:10:53,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e34be32 >2012-12-17 09:10:53,267 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [5d11909d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:10:53,460 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [570659e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:10:53,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e028116 >2012-12-17 09:10:53,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c78982f >2012-12-17 09:10:53,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:53,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, SpmStopVDSCommand, log id: c78982f >2012-12-17 09:10:53,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, ResetIrsVDSCommand, log id: 2e028116 >2012-12-17 09:10:53,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1028bcfe >2012-12-17 09:10:53,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:53,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1028bcfe >2012-12-17 09:10:53,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:10:53,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4774 >2012-12-17 09:10:53,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4774, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fca5d1e >2012-12-17 09:10:53,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling started: taskId = 21abe0df-6443-400b-91e6-e07a2146e7cf >2012-12-17 09:10:54,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended: taskId = 21abe0df-6443-400b-91e6-e07a2146e7cf task status = finished >2012-12-17 09:10:54,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended, spm status: SPM >2012-12-17 09:10:54,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=21abe0df-6443-400b-91e6-e07a2146e7cf), log id: 52f67846 >2012-12-17 09:10:54,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMClearTaskVDSCommand, log id: 52f67846 >2012-12-17 09:10:54,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33526637, log id: 2fca5d1e >2012-12-17 09:10:54,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:10:54,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:54,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c2707bc >2012-12-17 09:10:54,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c2707bc >2012-12-17 09:10:54,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e34be32 >2012-12-17 09:10:54,990 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:10:54,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14017cd >2012-12-17 09:10:54,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:10:54,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 96b4081 >2012-12-17 09:10:55,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 96b4081 >2012-12-17 09:10:55,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14017cd >2012-12-17 09:10:55,010 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) [4dd32292] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:11:03,868 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [4c00bcb5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:11:04,018 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:11:04,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 70119148 >2012-12-17 09:11:04,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63aaa867 >2012-12-17 09:11:04,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:04,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, SpmStopVDSCommand, log id: 63aaa867 >2012-12-17 09:11:04,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, ResetIrsVDSCommand, log id: 70119148 >2012-12-17 09:11:04,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 602175da >2012-12-17 09:11:04,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:04,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 602175da >2012-12-17 09:11:04,685 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:11:14,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:11:14,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4775 >2012-12-17 09:11:14,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4775, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52da58f1 >2012-12-17 09:11:14,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling started: taskId = 7fe6b14e-b819-41ca-a7a5-9c9dc758b06a >2012-12-17 09:11:15,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling ended: taskId = 7fe6b14e-b819-41ca-a7a5-9c9dc758b06a task status = finished >2012-12-17 09:11:15,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] spmStart polling ended, spm status: SPM >2012-12-17 09:11:15,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [388f9500] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7fe6b14e-b819-41ca-a7a5-9c9dc758b06a), log id: 378a6c >2012-12-17 09:11:15,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, HSMClearTaskVDSCommand, log id: 378a6c >2012-12-17 09:11:15,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [388f9500] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e60b890, log id: 52da58f1 >2012-12-17 09:11:16,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:11:16,622 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:11:16,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d2f3b33 >2012-12-17 09:11:16,679 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [388f9500] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:11:16,776 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [31ec2033] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:11:16,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4af22ae5 >2012-12-17 09:11:16,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 637f81bd >2012-12-17 09:11:16,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:17,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, SpmStopVDSCommand, log id: 637f81bd >2012-12-17 09:11:17,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, ResetIrsVDSCommand, log id: 4af22ae5 >2012-12-17 09:11:17,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 380a2afa >2012-12-17 09:11:17,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:17,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 380a2afa >2012-12-17 09:11:17,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:11:17,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4776 >2012-12-17 09:11:17,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4776, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c9f42e9 >2012-12-17 09:11:17,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling started: taskId = 1067e1b4-9c2b-4ed6-973c-42fab28f504f >2012-12-17 09:11:18,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended: taskId = 1067e1b4-9c2b-4ed6-973c-42fab28f504f task status = finished >2012-12-17 09:11:18,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended, spm status: SPM >2012-12-17 09:11:18,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1067e1b4-9c2b-4ed6-973c-42fab28f504f), log id: a007a9b >2012-12-17 09:11:18,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMClearTaskVDSCommand, log id: a007a9b >2012-12-17 09:11:18,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@25715b3e, log id: 7c9f42e9 >2012-12-17 09:11:18,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:11:18,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:18,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21cfae1b >2012-12-17 09:11:18,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21cfae1b >2012-12-17 09:11:18,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d2f3b33 >2012-12-17 09:11:18,785 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:11:18,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49af7ff4 >2012-12-17 09:11:18,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:18,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43ec14c9 >2012-12-17 09:11:18,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43ec14c9 >2012-12-17 09:11:18,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49af7ff4 >2012-12-17 09:11:18,805 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [63fee526] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:11:25,147 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:11:27,653 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [4557a90c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:11:28,047 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [2ed19e27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:11:28,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63fcd1d2 >2012-12-17 09:11:28,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 610168b6 >2012-12-17 09:11:28,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:28,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, SpmStopVDSCommand, log id: 610168b6 >2012-12-17 09:11:28,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, ResetIrsVDSCommand, log id: 63fcd1d2 >2012-12-17 09:11:28,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d162fa0 >2012-12-17 09:11:28,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2ed19e27] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:28,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2ed19e27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d162fa0 >2012-12-17 09:11:36,273 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:11:38,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:11:38,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4777 >2012-12-17 09:11:38,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4777, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f2edf66 >2012-12-17 09:11:38,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling started: taskId = 4b4b126d-41eb-4ebe-a356-62c3b9d38cbb >2012-12-17 09:11:39,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling ended: taskId = 4b4b126d-41eb-4ebe-a356-62c3b9d38cbb task status = finished >2012-12-17 09:11:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] spmStart polling ended, spm status: SPM >2012-12-17 09:11:39,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b4b126d-41eb-4ebe-a356-62c3b9d38cbb), log id: 515ba7c5 >2012-12-17 09:11:39,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, HSMClearTaskVDSCommand, log id: 515ba7c5 >2012-12-17 09:11:39,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [38fb24e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5863a03d, log id: f2edf66 >2012-12-17 09:11:39,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:11:39,990 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:11:39,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ad35453 >2012-12-17 09:11:40,026 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [38fb24e7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:11:40,327 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [733324dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:11:41,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 666d0c86 >2012-12-17 09:11:41,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b6d3b9b >2012-12-17 09:11:41,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:41,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SpmStopVDSCommand, log id: 6b6d3b9b >2012-12-17 09:11:41,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, ResetIrsVDSCommand, log id: 666d0c86 >2012-12-17 09:11:41,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fcd418d >2012-12-17 09:11:41,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:41,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fcd418d >2012-12-17 09:11:42,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:11:42,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4778 >2012-12-17 09:11:42,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4778, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d24de12 >2012-12-17 09:11:42,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = 8d264abd-4ea2-4d24-9025-e745e952686f >2012-12-17 09:11:43,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = 8d264abd-4ea2-4d24-9025-e745e952686f task status = finished >2012-12-17 09:11:43,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM >2012-12-17 09:11:43,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8d264abd-4ea2-4d24-9025-e745e952686f), log id: 3e4351be >2012-12-17 09:11:43,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: 3e4351be >2012-12-17 09:11:43,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35abb0e1, log id: 6d24de12 >2012-12-17 09:11:43,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:11:43,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:43,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a8df784 >2012-12-17 09:11:43,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a8df784 >2012-12-17 09:11:43,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ad35453 >2012-12-17 09:11:43,339 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:11:43,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2e752159 >2012-12-17 09:11:43,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:43,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1829f3b9 >2012-12-17 09:11:43,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1829f3b9 >2012-12-17 09:11:43,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e752159 >2012-12-17 09:11:43,359 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [39a5c05e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:11:46,816 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:11:52,140 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [455c512f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:11:52,356 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [13059b24] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:11:52,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bc5954a >2012-12-17 09:11:52,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f51c897 >2012-12-17 09:11:52,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:52,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, SpmStopVDSCommand, log id: 1f51c897 >2012-12-17 09:11:52,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, ResetIrsVDSCommand, log id: 5bc5954a >2012-12-17 09:11:52,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21724449 >2012-12-17 09:11:52,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:11:52,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21724449 >2012-12-17 09:11:57,113 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:12:02,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:02,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4779 >2012-12-17 09:12:02,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4779, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68be638 >2012-12-17 09:12:02,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling started: taskId = 593911e9-a3a4-4dfa-97b3-56a8f189bc25 >2012-12-17 09:12:03,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended: taskId = 593911e9-a3a4-4dfa-97b3-56a8f189bc25 task status = finished >2012-12-17 09:12:03,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] spmStart polling ended, spm status: SPM >2012-12-17 09:12:03,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=593911e9-a3a4-4dfa-97b3-56a8f189bc25), log id: cfb9b00 >2012-12-17 09:12:03,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, HSMClearTaskVDSCommand, log id: cfb9b00 >2012-12-17 09:12:03,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [214f1ee4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a7d23a8, log id: 68be638 >2012-12-17 09:12:03,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:04,006 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:12:04,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33c8a1d9 >2012-12-17 09:12:04,025 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [214f1ee4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:12:04,359 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [786aa7bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:12:04,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fa59a40 >2012-12-17 09:12:04,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74ad3655 >2012-12-17 09:12:04,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:04,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStopVDSCommand, log id: 74ad3655 >2012-12-17 09:12:04,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, ResetIrsVDSCommand, log id: 6fa59a40 >2012-12-17 09:12:04,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d1e303 >2012-12-17 09:12:04,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:04,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d1e303 >2012-12-17 09:12:04,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:05,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4780 >2012-12-17 09:12:05,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4780, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7fc380c2 >2012-12-17 09:12:05,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling started: taskId = 6777dc2e-4755-44e6-8ec8-ea7687983784 >2012-12-17 09:12:06,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling ended: taskId = 6777dc2e-4755-44e6-8ec8-ea7687983784 task status = finished >2012-12-17 09:12:06,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] spmStart polling ended, spm status: SPM >2012-12-17 09:12:06,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6777dc2e-4755-44e6-8ec8-ea7687983784), log id: 673d694d >2012-12-17 09:12:06,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMClearTaskVDSCommand, log id: 673d694d >2012-12-17 09:12:06,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70cc732e, log id: 7fc380c2 >2012-12-17 09:12:06,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:06,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:06,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27914022 >2012-12-17 09:12:06,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27914022 >2012-12-17 09:12:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33c8a1d9 >2012-12-17 09:12:06,844 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31098c79 >2012-12-17 09:12:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:06,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a77f86 >2012-12-17 09:12:06,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a77f86 >2012-12-17 09:12:06,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-93) [1d6be49e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31098c79 >2012-12-17 09:12:06,864 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [1d6be49e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:07,374 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:12:15,062 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5044c757] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:12:15,302 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [26c9f399] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:12:15,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f82833e >2012-12-17 09:12:15,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3216fe83 >2012-12-17 09:12:15,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:15,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SpmStopVDSCommand, log id: 3216fe83 >2012-12-17 09:12:16,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, ResetIrsVDSCommand, log id: 5f82833e >2012-12-17 09:12:16,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e0bff61 >2012-12-17 09:12:16,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:16,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e0bff61 >2012-12-17 09:12:17,828 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:12:26,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:26,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4781 >2012-12-17 09:12:26,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4781, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69b9d539 >2012-12-17 09:12:26,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling started: taskId = 29a28ae1-8018-47f5-9f54-48460a388266 >2012-12-17 09:12:27,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling ended: taskId = 29a28ae1-8018-47f5-9f54-48460a388266 task status = finished >2012-12-17 09:12:27,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] spmStart polling ended, spm status: SPM >2012-12-17 09:12:27,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [6573753] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=29a28ae1-8018-47f5-9f54-48460a388266), log id: 7ed923c3 >2012-12-17 09:12:27,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, HSMClearTaskVDSCommand, log id: 7ed923c3 >2012-12-17 09:12:27,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [6573753] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61bb1844, log id: 69b9d539 >2012-12-17 09:12:27,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:27,800 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:12:27,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33bc72bb >2012-12-17 09:12:27,820 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [6573753] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:12:27,989 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [40c8f45e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:12:28,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 67f6b57b >2012-12-17 09:12:28,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1aee0f0e >2012-12-17 09:12:28,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:28,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, SpmStopVDSCommand, log id: 1aee0f0e >2012-12-17 09:12:28,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, ResetIrsVDSCommand, log id: 67f6b57b >2012-12-17 09:12:28,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24a25395 >2012-12-17 09:12:28,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:28,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24a25395 >2012-12-17 09:12:28,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:28,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4782 >2012-12-17 09:12:28,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4782, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c2255d3 >2012-12-17 09:12:28,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling started: taskId = 628b1aca-6999-4280-a543-cb7a8efd2dfd >2012-12-17 09:12:29,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended: taskId = 628b1aca-6999-4280-a543-cb7a8efd2dfd task status = finished >2012-12-17 09:12:29,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended, spm status: SPM >2012-12-17 09:12:29,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=628b1aca-6999-4280-a543-cb7a8efd2dfd), log id: 185c3ea1 >2012-12-17 09:12:29,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMClearTaskVDSCommand, log id: 185c3ea1 >2012-12-17 09:12:29,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5013c885, log id: 2c2255d3 >2012-12-17 09:12:29,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:29,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:29,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b6a4df9 >2012-12-17 09:12:29,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b6a4df9 >2012-12-17 09:12:29,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33bc72bb >2012-12-17 09:12:29,572 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:29,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50252cb3 >2012-12-17 09:12:29,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:29,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76d9c659 >2012-12-17 09:12:29,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76d9c659 >2012-12-17 09:12:29,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50252cb3 >2012-12-17 09:12:29,593 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:38,415 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [72736c3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:12:38,749 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:12:38,895 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:12:39,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 58196ba3 >2012-12-17 09:12:39,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 66a12ca9 >2012-12-17 09:12:39,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:39,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, SpmStopVDSCommand, log id: 66a12ca9 >2012-12-17 09:12:39,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, ResetIrsVDSCommand, log id: 58196ba3 >2012-12-17 09:12:39,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 401b56a5 >2012-12-17 09:12:39,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:39,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 401b56a5 >2012-12-17 09:12:49,217 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:12:49,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:49,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4783 >2012-12-17 09:12:49,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4783, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d5b1bf >2012-12-17 09:12:50,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling started: taskId = 196efeca-10cc-469a-a117-96b410c6fc96 >2012-12-17 09:12:51,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling ended: taskId = 196efeca-10cc-469a-a117-96b410c6fc96 task status = finished >2012-12-17 09:12:51,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] spmStart polling ended, spm status: SPM >2012-12-17 09:12:51,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=196efeca-10cc-469a-a117-96b410c6fc96), log id: 322a17c2 >2012-12-17 09:12:51,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, HSMClearTaskVDSCommand, log id: 322a17c2 >2012-12-17 09:12:51,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [24cba1b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@697ddb48, log id: 4d5b1bf >2012-12-17 09:12:51,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:51,202 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:12:51,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21aea8d5 >2012-12-17 09:12:51,222 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [24cba1b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:12:51,537 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [3aebafd8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:12:51,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6617d0cc >2012-12-17 09:12:51,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 558b84aa >2012-12-17 09:12:51,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:51,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SpmStopVDSCommand, log id: 558b84aa >2012-12-17 09:12:51,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, ResetIrsVDSCommand, log id: 6617d0cc >2012-12-17 09:12:51,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59fbe422 >2012-12-17 09:12:51,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:51,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59fbe422 >2012-12-17 09:12:52,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:12:52,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4784 >2012-12-17 09:12:52,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4784, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a79eea3 >2012-12-17 09:12:52,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling started: taskId = 1e0b0139-1b4b-4a2e-9d49-53f366e033a9 >2012-12-17 09:12:53,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended: taskId = 1e0b0139-1b4b-4a2e-9d49-53f366e033a9 task status = finished >2012-12-17 09:12:53,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended, spm status: SPM >2012-12-17 09:12:53,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e0b0139-1b4b-4a2e-9d49-53f366e033a9), log id: 20987ee8 >2012-12-17 09:12:53,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMClearTaskVDSCommand, log id: 20987ee8 >2012-12-17 09:12:53,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bb3c20f, log id: 6a79eea3 >2012-12-17 09:12:53,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:12:53,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:53,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31814b62 >2012-12-17 09:12:53,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31814b62 >2012-12-17 09:12:53,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21aea8d5 >2012-12-17 09:12:53,426 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:53,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f7c2852 >2012-12-17 09:12:53,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:12:53,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 44262c84 >2012-12-17 09:12:53,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44262c84 >2012-12-17 09:12:53,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f7c2852 >2012-12-17 09:12:53,447 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-17) [3aea96b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:12:59,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:02,179 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7e0f01b0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:13:02,508 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [63fee8f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:13:02,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cf0d994 >2012-12-17 09:13:02,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: fc2d205 >2012-12-17 09:13:02,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:02,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, SpmStopVDSCommand, log id: fc2d205 >2012-12-17 09:13:03,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, ResetIrsVDSCommand, log id: 4cf0d994 >2012-12-17 09:13:03,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a76133e >2012-12-17 09:13:03,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:03,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a76133e >2012-12-17 09:13:09,883 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:13,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:13:13,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4785 >2012-12-17 09:13:13,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4785, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e887466 >2012-12-17 09:13:13,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling started: taskId = e9b1b346-78fd-4602-8f4a-9544e568f59b >2012-12-17 09:13:14,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended: taskId = e9b1b346-78fd-4602-8f4a-9544e568f59b task status = finished >2012-12-17 09:13:14,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] spmStart polling ended, spm status: SPM >2012-12-17 09:13:14,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9b1b346-78fd-4602-8f4a-9544e568f59b), log id: 43df756 >2012-12-17 09:13:14,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, HSMClearTaskVDSCommand, log id: 43df756 >2012-12-17 09:13:14,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [7d79337c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@393147, log id: 5e887466 >2012-12-17 09:13:14,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:13:14,418 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:13:14,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5330079d >2012-12-17 09:13:14,488 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [7d79337c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:13:14,846 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [66bf54ed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:13:15,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 38f85236 >2012-12-17 09:13:15,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 59bf952f >2012-12-17 09:13:15,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:15,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, SpmStopVDSCommand, log id: 59bf952f >2012-12-17 09:13:15,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, ResetIrsVDSCommand, log id: 38f85236 >2012-12-17 09:13:15,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c31b39c >2012-12-17 09:13:15,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [66bf54ed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:15,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [66bf54ed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c31b39c >2012-12-17 09:13:15,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:13:15,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4786 >2012-12-17 09:13:15,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4786, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ecb63f6 >2012-12-17 09:13:15,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = cb45693f-3eb3-4aad-9289-ab143e955389 >2012-12-17 09:13:16,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = cb45693f-3eb3-4aad-9289-ab143e955389 task status = finished >2012-12-17 09:13:16,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 09:13:16,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb45693f-3eb3-4aad-9289-ab143e955389), log id: 167dacb5 >2012-12-17 09:13:16,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 167dacb5 >2012-12-17 09:13:16,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14d5aa9c, log id: 5ecb63f6 >2012-12-17 09:13:16,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:13:16,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:16,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78be70e0 >2012-12-17 09:13:16,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78be70e0 >2012-12-17 09:13:16,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5330079d >2012-12-17 09:13:16,579 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:13:16,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ee4c00e >2012-12-17 09:13:16,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:16,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d59a879 >2012-12-17 09:13:16,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d59a879 >2012-12-17 09:13:16,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ee4c00e >2012-12-17 09:13:16,599 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:13:20,211 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:25,408 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [733172b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:13:26,260 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:13:26,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4575eb09 >2012-12-17 09:13:26,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bb0ccb2 >2012-12-17 09:13:26,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:26,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, SpmStopVDSCommand, log id: 3bb0ccb2 >2012-12-17 09:13:26,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, ResetIrsVDSCommand, log id: 4575eb09 >2012-12-17 09:13:26,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18c3de01 >2012-12-17 09:13:26,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:26,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18c3de01 >2012-12-17 09:13:31,535 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:36,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:13:36,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4787 >2012-12-17 09:13:36,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4787, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75772e6a >2012-12-17 09:13:36,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling started: taskId = 2b1fa8ab-7a28-4312-bb5e-59dde4310641 >2012-12-17 09:13:37,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling ended: taskId = 2b1fa8ab-7a28-4312-bb5e-59dde4310641 task status = finished >2012-12-17 09:13:37,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] spmStart polling ended, spm status: SPM >2012-12-17 09:13:37,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [480a798f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b1fa8ab-7a28-4312-bb5e-59dde4310641), log id: 5ce9013 >2012-12-17 09:13:37,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, HSMClearTaskVDSCommand, log id: 5ce9013 >2012-12-17 09:13:37,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [480a798f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c54d25f, log id: 75772e6a >2012-12-17 09:13:37,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:13:37,932 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:13:37,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 276c45e2 >2012-12-17 09:13:37,981 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [480a798f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:13:38,163 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [1f8f058a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:13:38,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 34a79e1 >2012-12-17 09:13:38,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75f7498e >2012-12-17 09:13:38,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:38,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, SpmStopVDSCommand, log id: 75f7498e >2012-12-17 09:13:38,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, ResetIrsVDSCommand, log id: 34a79e1 >2012-12-17 09:13:38,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a4175ae >2012-12-17 09:13:38,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:38,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a4175ae >2012-12-17 09:13:38,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:13:38,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4788 >2012-12-17 09:13:38,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4788, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20e6b4f7 >2012-12-17 09:13:38,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 6f598347-53cc-4719-9545-0f263ea5553f >2012-12-17 09:13:39,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 6f598347-53cc-4719-9545-0f263ea5553f task status = finished >2012-12-17 09:13:39,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM >2012-12-17 09:13:39,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f598347-53cc-4719-9545-0f263ea5553f), log id: 26c44711 >2012-12-17 09:13:39,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 26c44711 >2012-12-17 09:13:39,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55e1d7c3, log id: 20e6b4f7 >2012-12-17 09:13:39,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:13:39,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:39,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 33f7c17a >2012-12-17 09:13:39,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33f7c17a >2012-12-17 09:13:39,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 276c45e2 >2012-12-17 09:13:39,752 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:13:39,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ebf4b3d >2012-12-17 09:13:39,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:39,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c30c9ad >2012-12-17 09:13:39,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c30c9ad >2012-12-17 09:13:39,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ebf4b3d >2012-12-17 09:13:39,772 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:13:41,955 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:48,592 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [680d007a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:13:48,809 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [50cb2122] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:13:48,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e7d119f >2012-12-17 09:13:48,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 271e280c >2012-12-17 09:13:48,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:49,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, SpmStopVDSCommand, log id: 271e280c >2012-12-17 09:13:49,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, ResetIrsVDSCommand, log id: 1e7d119f >2012-12-17 09:13:49,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2238362e >2012-12-17 09:13:49,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50cb2122] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:13:49,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50cb2122] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2238362e >2012-12-17 09:13:52,248 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:13:59,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:13:59,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4789 >2012-12-17 09:13:59,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4789, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e5dc99c >2012-12-17 09:13:59,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling started: taskId = 93869726-582d-4305-9f04-4e69e3686a13 >2012-12-17 09:14:00,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended: taskId = 93869726-582d-4305-9f04-4e69e3686a13 task status = finished >2012-12-17 09:14:00,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] spmStart polling ended, spm status: SPM >2012-12-17 09:14:00,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93869726-582d-4305-9f04-4e69e3686a13), log id: 477eb5c4 >2012-12-17 09:14:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, HSMClearTaskVDSCommand, log id: 477eb5c4 >2012-12-17 09:14:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [261e40a1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2076d2dd, log id: 1e5dc99c >2012-12-17 09:14:00,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:01,391 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:14:01,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43d57c81 >2012-12-17 09:14:01,460 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [261e40a1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:14:01,698 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [1598e29f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:14:01,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 391fa870 >2012-12-17 09:14:01,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 554cf377 >2012-12-17 09:14:01,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:01,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStopVDSCommand, log id: 554cf377 >2012-12-17 09:14:02,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, ResetIrsVDSCommand, log id: 391fa870 >2012-12-17 09:14:02,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20008d8e >2012-12-17 09:14:02,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:02,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20008d8e >2012-12-17 09:14:02,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:14:02,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4790 >2012-12-17 09:14:02,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4790, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2244d5b0 >2012-12-17 09:14:02,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling started: taskId = d2dc295e-a750-40a4-9522-7419b6b0b812 >2012-12-17 09:14:03,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling ended: taskId = d2dc295e-a750-40a4-9522-7419b6b0b812 task status = finished >2012-12-17 09:14:03,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] spmStart polling ended, spm status: SPM >2012-12-17 09:14:03,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2dc295e-a750-40a4-9522-7419b6b0b812), log id: 7ffaa9c0 >2012-12-17 09:14:03,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMClearTaskVDSCommand, log id: 7ffaa9c0 >2012-12-17 09:14:03,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219ebde4, log id: 2244d5b0 >2012-12-17 09:14:03,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:03,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:03,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c5efca0 >2012-12-17 09:14:03,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c5efca0 >2012-12-17 09:14:03,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 43d57c81 >2012-12-17 09:14:03,274 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:03,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 301b72b8 >2012-12-17 09:14:03,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:03,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ffae45b >2012-12-17 09:14:03,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ffae45b >2012-12-17 09:14:03,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [68b3db6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 301b72b8 >2012-12-17 09:14:03,294 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [68b3db6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:12,184 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [32d89e5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:14:12,394 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [3078e69b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:14:12,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d80a534 >2012-12-17 09:14:12,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35dec7ea >2012-12-17 09:14:12,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:12,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, SpmStopVDSCommand, log id: 35dec7ea >2012-12-17 09:14:12,881 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:14:12,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, ResetIrsVDSCommand, log id: 3d80a534 >2012-12-17 09:14:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e43d58f >2012-12-17 09:14:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3078e69b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3078e69b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e43d58f >2012-12-17 09:14:23,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:14:23,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4791 >2012-12-17 09:14:23,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4791, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70fea370 >2012-12-17 09:14:23,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling started: taskId = 869fd056-c3fb-491c-ae62-2263f44afbcd >2012-12-17 09:14:24,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling ended: taskId = 869fd056-c3fb-491c-ae62-2263f44afbcd task status = finished >2012-12-17 09:14:24,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] spmStart polling ended, spm status: SPM >2012-12-17 09:14:24,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=869fd056-c3fb-491c-ae62-2263f44afbcd), log id: 7f0bfc87 >2012-12-17 09:14:24,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, HSMClearTaskVDSCommand, log id: 7f0bfc87 >2012-12-17 09:14:24,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61acfe5d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6bf0443a, log id: 70fea370 >2012-12-17 09:14:24,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:24,383 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:14:24,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 856c21b >2012-12-17 09:14:24,452 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61acfe5d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:14:24,779 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [11c9636b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:14:24,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63402df6 >2012-12-17 09:14:24,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42c6bee4 >2012-12-17 09:14:24,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:25,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, SpmStopVDSCommand, log id: 42c6bee4 >2012-12-17 09:14:25,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, ResetIrsVDSCommand, log id: 63402df6 >2012-12-17 09:14:25,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fd0fb57 >2012-12-17 09:14:25,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [11c9636b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:25,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [11c9636b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fd0fb57 >2012-12-17 09:14:25,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:14:25,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4792 >2012-12-17 09:14:25,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4792, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a6d19dc >2012-12-17 09:14:26,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 77b02066-465c-4f59-b354-742d1955e043 >2012-12-17 09:14:27,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 77b02066-465c-4f59-b354-742d1955e043 task status = finished >2012-12-17 09:14:27,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM >2012-12-17 09:14:27,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77b02066-465c-4f59-b354-742d1955e043), log id: 35421146 >2012-12-17 09:14:27,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 35421146 >2012-12-17 09:14:27,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47c25fb, log id: 5a6d19dc >2012-12-17 09:14:27,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:27,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:27,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5150aff >2012-12-17 09:14:27,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5150aff >2012-12-17 09:14:27,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 856c21b >2012-12-17 09:14:27,187 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:27,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2884edf4 >2012-12-17 09:14:27,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:27,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bd21b70 >2012-12-17 09:14:27,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bd21b70 >2012-12-17 09:14:27,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2884edf4 >2012-12-17 09:14:27,208 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:33,560 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:14:36,299 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [67020b95] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:14:36,777 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [4d69cef1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:14:36,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 155462b9 >2012-12-17 09:14:36,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1700448a >2012-12-17 09:14:36,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:37,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, SpmStopVDSCommand, log id: 1700448a >2012-12-17 09:14:37,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, ResetIrsVDSCommand, log id: 155462b9 >2012-12-17 09:14:37,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a281dca >2012-12-17 09:14:37,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4d69cef1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:37,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [4d69cef1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a281dca >2012-12-17 09:14:43,858 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:14:47,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:14:47,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4793 >2012-12-17 09:14:47,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4793, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77e3b752 >2012-12-17 09:14:47,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling started: taskId = af61d983-217d-422c-94e2-9521fc65877c >2012-12-17 09:14:49,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling ended: taskId = af61d983-217d-422c-94e2-9521fc65877c task status = finished >2012-12-17 09:14:49,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] spmStart polling ended, spm status: SPM >2012-12-17 09:14:49,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [46a80193] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af61d983-217d-422c-94e2-9521fc65877c), log id: 3e246a24 >2012-12-17 09:14:49,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, HSMClearTaskVDSCommand, log id: 3e246a24 >2012-12-17 09:14:49,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [46a80193] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48b4815b, log id: 77e3b752 >2012-12-17 09:14:49,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:49,107 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:14:49,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22759ef9 >2012-12-17 09:14:49,119 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [46a80193] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:14:49,228 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [1ebf597f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:14:49,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25bb04fc >2012-12-17 09:14:49,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 72856b38 >2012-12-17 09:14:49,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:50,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SpmStopVDSCommand, log id: 72856b38 >2012-12-17 09:14:50,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, ResetIrsVDSCommand, log id: 25bb04fc >2012-12-17 09:14:50,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a620693 >2012-12-17 09:14:50,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:50,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a620693 >2012-12-17 09:14:50,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:14:50,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4794 >2012-12-17 09:14:50,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4794, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55dc0475 >2012-12-17 09:14:50,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling started: taskId = 75e830d7-a31a-4ca1-8ac9-3df7b53abf81 >2012-12-17 09:14:51,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended: taskId = 75e830d7-a31a-4ca1-8ac9-3df7b53abf81 task status = finished >2012-12-17 09:14:51,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended, spm status: SPM >2012-12-17 09:14:51,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=75e830d7-a31a-4ca1-8ac9-3df7b53abf81), log id: 71987d0e >2012-12-17 09:14:51,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMClearTaskVDSCommand, log id: 71987d0e >2012-12-17 09:14:51,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@523a5c65, log id: 55dc0475 >2012-12-17 09:14:51,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:14:51,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:51,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2caac86c >2012-12-17 09:14:51,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2caac86c >2012-12-17 09:14:51,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22759ef9 >2012-12-17 09:14:51,413 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:51,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e5740d2 >2012-12-17 09:14:51,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:14:51,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16d52d8 >2012-12-17 09:14:51,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16d52d8 >2012-12-17 09:14:51,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e5740d2 >2012-12-17 09:14:51,434 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [1598e29f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:14:54,111 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Checking autorecoverable hosts >2012-12-17 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Autorecovering 0 hosts >2012-12-17 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Checking autorecoverable hosts done >2012-12-17 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [94e4f4d] Autorecovering storage domains is disabled, skipping >2012-12-17 09:15:00,246 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [7d3891a6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:15:00,358 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [50bce313] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:15:00,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1000885c >2012-12-17 09:15:00,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a31d914 >2012-12-17 09:15:00,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:01,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SpmStopVDSCommand, log id: 3a31d914 >2012-12-17 09:15:01,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, ResetIrsVDSCommand, log id: 1000885c >2012-12-17 09:15:01,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30a569d9 >2012-12-17 09:15:01,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:01,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30a569d9 >2012-12-17 09:15:04,317 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:15:11,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:15:11,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4795 >2012-12-17 09:15:11,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4795, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 41e8cfe9 >2012-12-17 09:15:11,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling started: taskId = 1713bbb0-ec21-4f86-bd97-dc8765138b74 >2012-12-17 09:15:12,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended: taskId = 1713bbb0-ec21-4f86-bd97-dc8765138b74 task status = finished >2012-12-17 09:15:12,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] spmStart polling ended, spm status: SPM >2012-12-17 09:15:12,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1713bbb0-ec21-4f86-bd97-dc8765138b74), log id: 315b8ccf >2012-12-17 09:15:13,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, HSMClearTaskVDSCommand, log id: 315b8ccf >2012-12-17 09:15:13,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [ccd5055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71798359, log id: 41e8cfe9 >2012-12-17 09:15:13,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:15:13,168 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:15:13,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c7f49bd >2012-12-17 09:15:13,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [ccd5055] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:15:13,324 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [e42ef5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:15:13,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 11d58cb8 >2012-12-17 09:15:13,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38e24b06 >2012-12-17 09:15:13,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:14,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, SpmStopVDSCommand, log id: 38e24b06 >2012-12-17 09:15:14,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, ResetIrsVDSCommand, log id: 11d58cb8 >2012-12-17 09:15:14,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6191f443 >2012-12-17 09:15:14,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [e42ef5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:14,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [e42ef5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6191f443 >2012-12-17 09:15:14,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:15:14,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4796 >2012-12-17 09:15:14,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4796, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c64674d >2012-12-17 09:15:14,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling started: taskId = de0dd12d-7442-4cfe-86d7-dc672c2fbc73 >2012-12-17 09:15:15,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling ended: taskId = de0dd12d-7442-4cfe-86d7-dc672c2fbc73 task status = finished >2012-12-17 09:15:15,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] spmStart polling ended, spm status: SPM >2012-12-17 09:15:15,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=de0dd12d-7442-4cfe-86d7-dc672c2fbc73), log id: 500a5409 >2012-12-17 09:15:15,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMClearTaskVDSCommand, log id: 500a5409 >2012-12-17 09:15:15,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@561343ec, log id: 3c64674d >2012-12-17 09:15:15,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:15:15,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:15,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ceff84 >2012-12-17 09:15:15,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ceff84 >2012-12-17 09:15:15,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c7f49bd >2012-12-17 09:15:15,327 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:15:15,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 378603c1 >2012-12-17 09:15:15,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:15,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e9392c9 >2012-12-17 09:15:15,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e9392c9 >2012-12-17 09:15:15,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [5231d6f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 378603c1 >2012-12-17 09:15:15,347 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [5231d6f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:15:24,198 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50cb2122] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:15:24,284 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [49192055] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:15:24,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 430c7df9 >2012-12-17 09:15:24,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 438cc746 >2012-12-17 09:15:24,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:24,699 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:15:25,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, SpmStopVDSCommand, log id: 438cc746 >2012-12-17 09:15:25,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, ResetIrsVDSCommand, log id: 430c7df9 >2012-12-17 09:15:25,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b59f8ac >2012-12-17 09:15:25,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:25,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b59f8ac >2012-12-17 09:15:29,746 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [1940bae2] Running command: LogoutUserCommand internal: false. >2012-12-17 09:15:35,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:15:35,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4797 >2012-12-17 09:15:35,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4797, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67f1aa2c >2012-12-17 09:15:35,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling started: taskId = 42ca25d3-9536-4de7-91be-4cc049204a87 >2012-12-17 09:15:36,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended: taskId = 42ca25d3-9536-4de7-91be-4cc049204a87 task status = finished >2012-12-17 09:15:36,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] spmStart polling ended, spm status: SPM >2012-12-17 09:15:36,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=42ca25d3-9536-4de7-91be-4cc049204a87), log id: 368e75d9 >2012-12-17 09:15:36,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, HSMClearTaskVDSCommand, log id: 368e75d9 >2012-12-17 09:15:36,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [55d82264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42db158a, log id: 67f1aa2c >2012-12-17 09:15:36,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:15:36,541 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:15:36,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 495d0c82 >2012-12-17 09:15:36,557 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [55d82264] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:15:36,692 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [6c5669f4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:15:36,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77c52c29 >2012-12-17 09:15:36,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18d4f5a6 >2012-12-17 09:15:36,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:36,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, SpmStopVDSCommand, log id: 18d4f5a6 >2012-12-17 09:15:36,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, ResetIrsVDSCommand, log id: 77c52c29 >2012-12-17 09:15:37,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7be18433 >2012-12-17 09:15:37,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:37,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7be18433 >2012-12-17 09:15:37,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:15:37,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4798 >2012-12-17 09:15:37,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4798, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26f9e564 >2012-12-17 09:15:37,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = 35d60402-bc3b-489f-add5-9240cfe8a894 >2012-12-17 09:15:38,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = 35d60402-bc3b-489f-add5-9240cfe8a894 task status = finished >2012-12-17 09:15:38,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM >2012-12-17 09:15:38,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35d60402-bc3b-489f-add5-9240cfe8a894), log id: 14603702 >2012-12-17 09:15:38,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 14603702 >2012-12-17 09:15:38,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38bb6ec1, log id: 26f9e564 >2012-12-17 09:15:38,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:15:38,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:38,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51941608 >2012-12-17 09:15:38,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51941608 >2012-12-17 09:15:38,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 495d0c82 >2012-12-17 09:15:38,271 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:15:38,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ed1996e >2012-12-17 09:15:38,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:38,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 423d9c0b >2012-12-17 09:15:38,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 423d9c0b >2012-12-17 09:15:38,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ed1996e >2012-12-17 09:15:38,291 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:15:45,136 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:15:47,116 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5bd08b5f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:15:47,260 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [efbb817] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:15:47,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 539fd57 >2012-12-17 09:15:47,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c38f909 >2012-12-17 09:15:47,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:47,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, SpmStopVDSCommand, log id: 6c38f909 >2012-12-17 09:15:48,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, ResetIrsVDSCommand, log id: 539fd57 >2012-12-17 09:15:48,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [efbb817] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b6a8121 >2012-12-17 09:15:48,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:15:48,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [efbb817] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b6a8121 >2012-12-17 09:15:56,330 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:15:58,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:15:58,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4799 >2012-12-17 09:15:58,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4799, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6df7dc86 >2012-12-17 09:15:58,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling started: taskId = d584a788-8d17-451e-a8e5-0acc8b848d2d >2012-12-17 09:16:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended: taskId = d584a788-8d17-451e-a8e5-0acc8b848d2d task status = finished >2012-12-17 09:16:00,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] spmStart polling ended, spm status: SPM >2012-12-17 09:16:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d584a788-8d17-451e-a8e5-0acc8b848d2d), log id: 6da268f9 >2012-12-17 09:16:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, HSMClearTaskVDSCommand, log id: 6da268f9 >2012-12-17 09:16:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7b0d0943] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4199ebaa, log id: 6df7dc86 >2012-12-17 09:16:00,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:00,309 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:16:00,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cf61f10 >2012-12-17 09:16:00,324 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7b0d0943] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:01,329 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:01,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3bb25634 >2012-12-17 09:16:01,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4305d5e0 >2012-12-17 09:16:01,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:01,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, SpmStopVDSCommand, log id: 4305d5e0 >2012-12-17 09:16:01,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, ResetIrsVDSCommand, log id: 3bb25634 >2012-12-17 09:16:01,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 727efebd >2012-12-17 09:16:01,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:01,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 727efebd >2012-12-17 09:16:01,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:16:01,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4800 >2012-12-17 09:16:01,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4800, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3785312d >2012-12-17 09:16:02,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling started: taskId = af9d8663-9fde-4f7a-b238-4492ae57d288 >2012-12-17 09:16:03,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended: taskId = af9d8663-9fde-4f7a-b238-4492ae57d288 task status = finished >2012-12-17 09:16:03,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended, spm status: SPM >2012-12-17 09:16:03,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af9d8663-9fde-4f7a-b238-4492ae57d288), log id: 480ce5de >2012-12-17 09:16:03,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMClearTaskVDSCommand, log id: 480ce5de >2012-12-17 09:16:03,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46c776c6, log id: 3785312d >2012-12-17 09:16:03,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:03,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:03,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ec71a4f >2012-12-17 09:16:03,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ec71a4f >2012-12-17 09:16:03,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cf61f10 >2012-12-17 09:16:03,163 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:03,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5aca20 >2012-12-17 09:16:03,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:03,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29926e84 >2012-12-17 09:16:03,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29926e84 >2012-12-17 09:16:03,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5aca20 >2012-12-17 09:16:03,184 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:06,608 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:16:11,984 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [58c21a2d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:12,094 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [67dedb1b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:12,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3c090cef >2012-12-17 09:16:12,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1ad4d296 >2012-12-17 09:16:12,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:12,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SpmStopVDSCommand, log id: 1ad4d296 >2012-12-17 09:16:12,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, ResetIrsVDSCommand, log id: 3c090cef >2012-12-17 09:16:12,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 657baea5 >2012-12-17 09:16:12,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:12,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 657baea5 >2012-12-17 09:16:16,820 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:16:22,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:16:22,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4801 >2012-12-17 09:16:22,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4801, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 231f5bd7 >2012-12-17 09:16:22,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling started: taskId = c768d2b2-2ab1-4ec8-80a3-825b787f4da1 >2012-12-17 09:16:23,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling ended: taskId = c768d2b2-2ab1-4ec8-80a3-825b787f4da1 task status = finished >2012-12-17 09:16:23,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] spmStart polling ended, spm status: SPM >2012-12-17 09:16:23,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c768d2b2-2ab1-4ec8-80a3-825b787f4da1), log id: 5d1c94c2 >2012-12-17 09:16:23,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, HSMClearTaskVDSCommand, log id: 5d1c94c2 >2012-12-17 09:16:23,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [58ac9a0f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7f6a7c4f, log id: 231f5bd7 >2012-12-17 09:16:23,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:23,725 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:16:23,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 329468ee >2012-12-17 09:16:23,741 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [58ac9a0f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:23,892 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [5654bc9d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:23,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50797fcb >2012-12-17 09:16:23,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1a1f97cb >2012-12-17 09:16:23,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:24,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStopVDSCommand, log id: 1a1f97cb >2012-12-17 09:16:24,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, ResetIrsVDSCommand, log id: 50797fcb >2012-12-17 09:16:24,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b868784 >2012-12-17 09:16:24,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:24,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b868784 >2012-12-17 09:16:24,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:16:24,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4802 >2012-12-17 09:16:24,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4802, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 596b069f >2012-12-17 09:16:24,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = 455bd6b6-e06c-4b43-ac16-8663aab41a02 >2012-12-17 09:16:25,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = 455bd6b6-e06c-4b43-ac16-8663aab41a02 task status = finished >2012-12-17 09:16:26,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM >2012-12-17 09:16:26,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=455bd6b6-e06c-4b43-ac16-8663aab41a02), log id: 12c939e8 >2012-12-17 09:16:26,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 12c939e8 >2012-12-17 09:16:26,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58f8b7fa, log id: 596b069f >2012-12-17 09:16:26,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:26,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:26,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b842bfc >2012-12-17 09:16:26,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b842bfc >2012-12-17 09:16:26,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 329468ee >2012-12-17 09:16:26,215 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:26,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 263bb81 >2012-12-17 09:16:26,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:26,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d22488e >2012-12-17 09:16:26,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d22488e >2012-12-17 09:16:26,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 263bb81 >2012-12-17 09:16:26,235 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-11) [2d4b4fe5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:27,122 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:16:34,993 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [436410e6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:35,150 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:35,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e06c90d >2012-12-17 09:16:35,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65633bc6 >2012-12-17 09:16:35,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:35,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, SpmStopVDSCommand, log id: 65633bc6 >2012-12-17 09:16:36,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, ResetIrsVDSCommand, log id: 1e06c90d >2012-12-17 09:16:36,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54d2bb05 >2012-12-17 09:16:36,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:36,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54d2bb05 >2012-12-17 09:16:37,489 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:16:46,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:16:46,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4803 >2012-12-17 09:16:46,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4803, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ff4f0cd >2012-12-17 09:16:46,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling started: taskId = 4aeafca6-cc76-4cb1-b7fe-45bb3a1229af >2012-12-17 09:16:47,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling ended: taskId = 4aeafca6-cc76-4cb1-b7fe-45bb3a1229af task status = finished >2012-12-17 09:16:47,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] spmStart polling ended, spm status: SPM >2012-12-17 09:16:47,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4aeafca6-cc76-4cb1-b7fe-45bb3a1229af), log id: 653e1d40 >2012-12-17 09:16:47,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, HSMClearTaskVDSCommand, log id: 653e1d40 >2012-12-17 09:16:47,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [1f8f058a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50f288ad, log id: ff4f0cd >2012-12-17 09:16:48,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:48,052 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:16:48,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4576353a >2012-12-17 09:16:48,131 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1f8f058a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:48,375 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:48,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b484266 >2012-12-17 09:16:48,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26e048f8 >2012-12-17 09:16:48,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:48,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, SpmStopVDSCommand, log id: 26e048f8 >2012-12-17 09:16:48,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, ResetIrsVDSCommand, log id: 6b484266 >2012-12-17 09:16:48,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 553ba559 >2012-12-17 09:16:48,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:48,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 553ba559 >2012-12-17 09:16:48,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:16:48,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4804 >2012-12-17 09:16:48,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4804, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4017204f >2012-12-17 09:16:48,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 9f8a611d-ef58-48d9-b324-eb4c69027521 >2012-12-17 09:16:49,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 9f8a611d-ef58-48d9-b324-eb4c69027521 task status = finished >2012-12-17 09:16:49,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM >2012-12-17 09:16:49,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9f8a611d-ef58-48d9-b324-eb4c69027521), log id: 51e076f1 >2012-12-17 09:16:49,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 51e076f1 >2012-12-17 09:16:49,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67a0f235, log id: 4017204f >2012-12-17 09:16:49,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:16:49,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:49,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 675af968 >2012-12-17 09:16:49,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 675af968 >2012-12-17 09:16:49,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4576353a >2012-12-17 09:16:49,960 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:49,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e58956a >2012-12-17 09:16:49,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:49,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55164841 >2012-12-17 09:16:49,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55164841 >2012-12-17 09:16:49,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e58956a >2012-12-17 09:16:49,982 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:16:57,879 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:16:58,827 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [af8a779] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:16:58,947 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [327be49e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:16:59,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f39e29d >2012-12-17 09:16:59,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39b8c902 >2012-12-17 09:16:59,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:59,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, SpmStopVDSCommand, log id: 39b8c902 >2012-12-17 09:16:59,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, ResetIrsVDSCommand, log id: 3f39e29d >2012-12-17 09:16:59,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [327be49e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a15dcb5 >2012-12-17 09:16:59,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:16:59,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [327be49e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a15dcb5 >2012-12-17 09:17:08,129 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:17:09,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:09,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4805 >2012-12-17 09:17:09,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4805, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21303019 >2012-12-17 09:17:09,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling started: taskId = 8551fddc-4665-41e6-8742-fa0217fafcaa >2012-12-17 09:17:10,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended: taskId = 8551fddc-4665-41e6-8742-fa0217fafcaa task status = finished >2012-12-17 09:17:10,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] spmStart polling ended, spm status: SPM >2012-12-17 09:17:10,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8551fddc-4665-41e6-8742-fa0217fafcaa), log id: 653f8f50 >2012-12-17 09:17:10,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, HSMClearTaskVDSCommand, log id: 653f8f50 >2012-12-17 09:17:10,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [4fb47a41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fda011e, log id: 21303019 >2012-12-17 09:17:10,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:10,411 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:17:10,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5430d7ac >2012-12-17 09:17:10,789 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [4fb47a41] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:17:12,084 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [5f63eb29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:17:12,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6b8762cf >2012-12-17 09:17:12,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 584ed778 >2012-12-17 09:17:12,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:12,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, SpmStopVDSCommand, log id: 584ed778 >2012-12-17 09:17:12,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, ResetIrsVDSCommand, log id: 6b8762cf >2012-12-17 09:17:12,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cc808a9 >2012-12-17 09:17:12,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5f63eb29] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:12,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5f63eb29] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cc808a9 >2012-12-17 09:17:12,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:12,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4806 >2012-12-17 09:17:12,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4806, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 152ff288 >2012-12-17 09:17:12,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling started: taskId = e9a2c215-5c17-4089-a5f0-e63ec7af7f32 >2012-12-17 09:17:13,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling ended: taskId = e9a2c215-5c17-4089-a5f0-e63ec7af7f32 task status = finished >2012-12-17 09:17:13,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] spmStart polling ended, spm status: SPM >2012-12-17 09:17:13,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9a2c215-5c17-4089-a5f0-e63ec7af7f32), log id: 72494f20 >2012-12-17 09:17:13,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMClearTaskVDSCommand, log id: 72494f20 >2012-12-17 09:17:13,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ae9f87e, log id: 152ff288 >2012-12-17 09:17:13,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:13,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:13,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 746253d3 >2012-12-17 09:17:13,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 746253d3 >2012-12-17 09:17:13,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5430d7ac >2012-12-17 09:17:13,718 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:17:13,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 636d7d56 >2012-12-17 09:17:13,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:13,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b42e7ac >2012-12-17 09:17:13,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b42e7ac >2012-12-17 09:17:13,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [1ebf597f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 636d7d56 >2012-12-17 09:17:13,739 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [1ebf597f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:17:18,369 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:17:22,543 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3078e69b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:17:22,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1240becc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:17:22,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 43eb0c5b >2012-12-17 09:17:22,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 52832e7 >2012-12-17 09:17:22,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:22,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStopVDSCommand, log id: 52832e7 >2012-12-17 09:17:22,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, ResetIrsVDSCommand, log id: 43eb0c5b >2012-12-17 09:17:22,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76b685ae >2012-12-17 09:17:22,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:23,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76b685ae >2012-12-17 09:17:28,586 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:17:33,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:33,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4807 >2012-12-17 09:17:33,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4807, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47685135 >2012-12-17 09:17:33,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling started: taskId = d4f8dba9-02ff-493d-aa62-37394b780ffd >2012-12-17 09:17:34,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended: taskId = d4f8dba9-02ff-493d-aa62-37394b780ffd task status = finished >2012-12-17 09:17:34,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] spmStart polling ended, spm status: SPM >2012-12-17 09:17:34,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d4f8dba9-02ff-493d-aa62-37394b780ffd), log id: 7fea60ce >2012-12-17 09:17:34,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, HSMClearTaskVDSCommand, log id: 7fea60ce >2012-12-17 09:17:34,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [b3b23b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7479b5fd, log id: 47685135 >2012-12-17 09:17:34,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:34,243 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:17:34,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 156ba855 >2012-12-17 09:17:34,260 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [b3b23b2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:17:34,363 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [2a7b65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:17:34,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44b8746a >2012-12-17 09:17:34,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24c53e62 >2012-12-17 09:17:34,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:34,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, SpmStopVDSCommand, log id: 24c53e62 >2012-12-17 09:17:34,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, ResetIrsVDSCommand, log id: 44b8746a >2012-12-17 09:17:34,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42aa18c3 >2012-12-17 09:17:34,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:34,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42aa18c3 >2012-12-17 09:17:34,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:34,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4808 >2012-12-17 09:17:34,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4808, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2256e3b9 >2012-12-17 09:17:34,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = c037311e-2cff-4bc5-b95d-d7605a169bf0 >2012-12-17 09:17:35,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = c037311e-2cff-4bc5-b95d-d7605a169bf0 task status = finished >2012-12-17 09:17:35,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM >2012-12-17 09:17:35,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c037311e-2cff-4bc5-b95d-d7605a169bf0), log id: 544577dc >2012-12-17 09:17:35,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 544577dc >2012-12-17 09:17:35,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1cc037bd, log id: 2256e3b9 >2012-12-17 09:17:36,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:36,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:36,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 638d4741 >2012-12-17 09:17:36,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 638d4741 >2012-12-17 09:17:36,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 156ba855 >2012-12-17 09:17:36,696 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:17:36,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74bd81f9 >2012-12-17 09:17:36,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:36,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 76214961 >2012-12-17 09:17:36,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 76214961 >2012-12-17 09:17:36,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74bd81f9 >2012-12-17 09:17:36,716 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [7f190dc3] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:17:38,807 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:17:44,792 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [37630ba1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:17:44,990 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [95f0e17] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:17:45,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 51a45297 >2012-12-17 09:17:45,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3c1f6d48 >2012-12-17 09:17:45,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:45,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, SpmStopVDSCommand, log id: 3c1f6d48 >2012-12-17 09:17:45,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, ResetIrsVDSCommand, log id: 51a45297 >2012-12-17 09:17:45,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70f34122 >2012-12-17 09:17:45,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:45,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [95f0e17] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70f34122 >2012-12-17 09:17:49,017 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:17:55,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:56,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4809 >2012-12-17 09:17:56,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4809, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44563f52 >2012-12-17 09:17:56,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling started: taskId = 97e5e958-f92b-4e4f-a830-f3a1b468792e >2012-12-17 09:17:57,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling ended: taskId = 97e5e958-f92b-4e4f-a830-f3a1b468792e task status = finished >2012-12-17 09:17:57,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] spmStart polling ended, spm status: SPM >2012-12-17 09:17:57,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=97e5e958-f92b-4e4f-a830-f3a1b468792e), log id: 779f176e >2012-12-17 09:17:57,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, HSMClearTaskVDSCommand, log id: 779f176e >2012-12-17 09:17:57,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7d47f958] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6dbb741c, log id: 44563f52 >2012-12-17 09:17:57,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:57,308 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:17:57,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e49064d >2012-12-17 09:17:57,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d47f958] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:17:57,561 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [7e80f448] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:17:57,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 742e020f >2012-12-17 09:17:57,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b7caae7 >2012-12-17 09:17:57,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:58,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, SpmStopVDSCommand, log id: 3b7caae7 >2012-12-17 09:17:58,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, ResetIrsVDSCommand, log id: 742e020f >2012-12-17 09:17:58,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56cb197f >2012-12-17 09:17:58,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:58,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56cb197f >2012-12-17 09:17:58,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:17:58,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4810 >2012-12-17 09:17:58,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4810, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44c76623 >2012-12-17 09:17:58,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling started: taskId = 625795c6-6075-4453-808a-4f0010ef2e37 >2012-12-17 09:17:59,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended: taskId = 625795c6-6075-4453-808a-4f0010ef2e37 task status = finished >2012-12-17 09:17:59,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended, spm status: SPM >2012-12-17 09:17:59,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=625795c6-6075-4453-808a-4f0010ef2e37), log id: 5dae82bd >2012-12-17 09:17:59,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMClearTaskVDSCommand, log id: 5dae82bd >2012-12-17 09:17:59,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c097657, log id: 44c76623 >2012-12-17 09:17:59,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:17:59,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:59,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dd6be65 >2012-12-17 09:17:59,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dd6be65 >2012-12-17 09:17:59,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e49064d >2012-12-17 09:17:59,618 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:17:59,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6ddffc >2012-12-17 09:17:59,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:17:59,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 616c55f9 >2012-12-17 09:17:59,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 616c55f9 >2012-12-17 09:17:59,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6ddffc >2012-12-17 09:17:59,638 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-68) [1756fb88] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:18:08,530 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1000cccb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:18:08,706 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:18:08,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7da4b87 >2012-12-17 09:18:08,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11be23a5 >2012-12-17 09:18:08,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:08,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, SpmStopVDSCommand, log id: 11be23a5 >2012-12-17 09:18:08,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, ResetIrsVDSCommand, log id: 7da4b87 >2012-12-17 09:18:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67c72dd1 >2012-12-17 09:18:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:08,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67c72dd1 >2012-12-17 09:18:09,637 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:18:19,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:18:19,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4811 >2012-12-17 09:18:19,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4811, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1eae0c7f >2012-12-17 09:18:19,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling started: taskId = 55613c4e-389e-4eb8-a501-cee38816843c >2012-12-17 09:18:20,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling ended: taskId = 55613c4e-389e-4eb8-a501-cee38816843c task status = finished >2012-12-17 09:18:20,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] spmStart polling ended, spm status: SPM >2012-12-17 09:18:20,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=55613c4e-389e-4eb8-a501-cee38816843c), log id: 6280e3f3 >2012-12-17 09:18:20,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, HSMClearTaskVDSCommand, log id: 6280e3f3 >2012-12-17 09:18:20,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3e5f7d32] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bf5990a, log id: 1eae0c7f >2012-12-17 09:18:20,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:18:20,244 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:18:20,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 98334b7 >2012-12-17 09:18:20,263 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3e5f7d32] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:18:20,392 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [20c0de80] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:18:20,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7bea430f >2012-12-17 09:18:20,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d9b0b2b >2012-12-17 09:18:20,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:21,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SpmStopVDSCommand, log id: 1d9b0b2b >2012-12-17 09:18:21,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, ResetIrsVDSCommand, log id: 7bea430f >2012-12-17 09:18:21,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1578c314 >2012-12-17 09:18:21,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:21,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1578c314 >2012-12-17 09:18:21,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:18:21,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4812 >2012-12-17 09:18:21,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4812, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e5e2160 >2012-12-17 09:18:21,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling started: taskId = 83c3a4d5-895e-47c5-a3ea-fe2be5d44be3 >2012-12-17 09:18:22,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling ended: taskId = 83c3a4d5-895e-47c5-a3ea-fe2be5d44be3 task status = finished >2012-12-17 09:18:22,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] spmStart polling ended, spm status: SPM >2012-12-17 09:18:22,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=83c3a4d5-895e-47c5-a3ea-fe2be5d44be3), log id: 49f01313 >2012-12-17 09:18:22,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMClearTaskVDSCommand, log id: 49f01313 >2012-12-17 09:18:22,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30746a51, log id: 5e5e2160 >2012-12-17 09:18:22,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:18:22,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:22,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 401c7134 >2012-12-17 09:18:22,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 401c7134 >2012-12-17 09:18:22,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 98334b7 >2012-12-17 09:18:22,421 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:18:22,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b4c2dce >2012-12-17 09:18:22,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:22,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fd279d6 >2012-12-17 09:18:22,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fd279d6 >2012-12-17 09:18:22,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [746c8fa6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b4c2dce >2012-12-17 09:18:22,442 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [746c8fa6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:18:30,016 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:18:31,395 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1d6be49e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:18:31,837 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [2dc4111d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:18:31,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ec6d613 >2012-12-17 09:18:31,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d4484b >2012-12-17 09:18:31,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:32,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, SpmStopVDSCommand, log id: 3d4484b >2012-12-17 09:18:32,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, ResetIrsVDSCommand, log id: 3ec6d613 >2012-12-17 09:18:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 708feda6 >2012-12-17 09:18:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 708feda6 >2012-12-17 09:18:40,263 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:18:42,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:18:42,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4813 >2012-12-17 09:18:42,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4813, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bf6361 >2012-12-17 09:18:42,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling started: taskId = 345bcfb3-b639-4b6a-b53e-704af0fd87f1 >2012-12-17 09:18:43,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling ended: taskId = 345bcfb3-b639-4b6a-b53e-704af0fd87f1 task status = finished >2012-12-17 09:18:43,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] spmStart polling ended, spm status: SPM >2012-12-17 09:18:43,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=345bcfb3-b639-4b6a-b53e-704af0fd87f1), log id: 3c2ca8cc >2012-12-17 09:18:43,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, HSMClearTaskVDSCommand, log id: 3c2ca8cc >2012-12-17 09:18:43,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [73cd7795] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17fe07f1, log id: 2bf6361 >2012-12-17 09:18:43,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:18:43,614 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:18:43,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 196cd471 >2012-12-17 09:18:43,638 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [73cd7795] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:18:43,792 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [76c72cb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:18:43,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7006cad7 >2012-12-17 09:18:43,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2b9b0a23 >2012-12-17 09:18:43,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:44,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, SpmStopVDSCommand, log id: 2b9b0a23 >2012-12-17 09:18:44,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, ResetIrsVDSCommand, log id: 7006cad7 >2012-12-17 09:18:44,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e9e4b0b >2012-12-17 09:18:44,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:44,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e9e4b0b >2012-12-17 09:18:44,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:18:44,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4814 >2012-12-17 09:18:44,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4814, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f5062f6 >2012-12-17 09:18:44,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = c8916c00-ccfe-433a-8344-149588056a1c >2012-12-17 09:18:45,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = c8916c00-ccfe-433a-8344-149588056a1c task status = finished >2012-12-17 09:18:45,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM >2012-12-17 09:18:45,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c8916c00-ccfe-433a-8344-149588056a1c), log id: 778d672d >2012-12-17 09:18:45,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: 778d672d >2012-12-17 09:18:45,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6470f8ac, log id: 6f5062f6 >2012-12-17 09:18:45,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:18:45,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:45,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b05fa49 >2012-12-17 09:18:45,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b05fa49 >2012-12-17 09:18:45,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 196cd471 >2012-12-17 09:18:45,300 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:18:45,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2572dc7b >2012-12-17 09:18:45,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:45,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 38921fb7 >2012-12-17 09:18:45,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 38921fb7 >2012-12-17 09:18:45,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2572dc7b >2012-12-17 09:18:45,320 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2155afb6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:18:51,405 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:18:54,156 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [75aa641b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:18:54,317 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [32133550] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:18:54,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 597b5eac >2012-12-17 09:18:54,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 12a035d0 >2012-12-17 09:18:54,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:55,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStopVDSCommand, log id: 12a035d0 >2012-12-17 09:18:55,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, ResetIrsVDSCommand, log id: 597b5eac >2012-12-17 09:18:55,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f8b91c7 >2012-12-17 09:18:55,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:18:55,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f8b91c7 >2012-12-17 09:19:01,806 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:19:05,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:05,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4815 >2012-12-17 09:19:05,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4815, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fd7f7a4 >2012-12-17 09:19:05,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling started: taskId = 0dcda5e5-0784-4c37-b054-2bb935ba9f67 >2012-12-17 09:19:06,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling ended: taskId = 0dcda5e5-0784-4c37-b054-2bb935ba9f67 task status = finished >2012-12-17 09:19:06,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] spmStart polling ended, spm status: SPM >2012-12-17 09:19:06,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [13059b24] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0dcda5e5-0784-4c37-b054-2bb935ba9f67), log id: 717e54d >2012-12-17 09:19:06,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, HSMClearTaskVDSCommand, log id: 717e54d >2012-12-17 09:19:06,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [13059b24] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31cf25de, log id: 4fd7f7a4 >2012-12-17 09:19:06,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:06,404 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:19:06,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 422f053c >2012-12-17 09:19:06,461 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [13059b24] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:19:06,755 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [1091267a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:19:06,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d89a5b5 >2012-12-17 09:19:06,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 65cff4a >2012-12-17 09:19:06,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:07,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, SpmStopVDSCommand, log id: 65cff4a >2012-12-17 09:19:07,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, ResetIrsVDSCommand, log id: 3d89a5b5 >2012-12-17 09:19:07,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1091267a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d35054b >2012-12-17 09:19:07,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1091267a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:07,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1091267a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d35054b >2012-12-17 09:19:07,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4816 >2012-12-17 09:19:07,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4816, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2637f7d8 >2012-12-17 09:19:07,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling started: taskId = da3593b6-fd6c-497a-b826-551485e178bf >2012-12-17 09:19:08,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended: taskId = da3593b6-fd6c-497a-b826-551485e178bf task status = finished >2012-12-17 09:19:08,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended, spm status: SPM >2012-12-17 09:19:08,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da3593b6-fd6c-497a-b826-551485e178bf), log id: 128ee247 >2012-12-17 09:19:08,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMClearTaskVDSCommand, log id: 128ee247 >2012-12-17 09:19:08,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62a9acd5, log id: 2637f7d8 >2012-12-17 09:19:08,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:08,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:08,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3390cc8c >2012-12-17 09:19:08,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3390cc8c >2012-12-17 09:19:08,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 422f053c >2012-12-17 09:19:08,286 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:19:08,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d78af32 >2012-12-17 09:19:08,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:08,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65196d6a >2012-12-17 09:19:08,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65196d6a >2012-12-17 09:19:08,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d78af32 >2012-12-17 09:19:08,307 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-36) [3ff8505] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:19:12,042 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:19:17,170 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [492f1d2b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:19:17,303 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [1513ca5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:19:17,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25089593 >2012-12-17 09:19:17,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ba1e543 >2012-12-17 09:19:17,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:17,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, SpmStopVDSCommand, log id: 5ba1e543 >2012-12-17 09:19:17,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, ResetIrsVDSCommand, log id: 25089593 >2012-12-17 09:19:17,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25b44d10 >2012-12-17 09:19:17,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:17,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25b44d10 >2012-12-17 09:19:22,263 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:19:27,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:27,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4817 >2012-12-17 09:19:27,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4817, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7154625c >2012-12-17 09:19:27,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling started: taskId = 22c829c2-c0b1-4633-b6d4-7950cc4c77df >2012-12-17 09:19:28,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling ended: taskId = 22c829c2-c0b1-4633-b6d4-7950cc4c77df task status = finished >2012-12-17 09:19:28,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] spmStart polling ended, spm status: SPM >2012-12-17 09:19:28,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [71efd086] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=22c829c2-c0b1-4633-b6d4-7950cc4c77df), log id: 36817ba >2012-12-17 09:19:28,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, HSMClearTaskVDSCommand, log id: 36817ba >2012-12-17 09:19:28,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [71efd086] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26edb615, log id: 7154625c >2012-12-17 09:19:28,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:28,933 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:19:28,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c35f8c7 >2012-12-17 09:19:28,964 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [71efd086] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:19:29,058 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:19:29,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22eb571b >2012-12-17 09:19:29,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 365064dd >2012-12-17 09:19:29,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:29,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, SpmStopVDSCommand, log id: 365064dd >2012-12-17 09:19:29,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, ResetIrsVDSCommand, log id: 22eb571b >2012-12-17 09:19:29,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c226ee4 >2012-12-17 09:19:29,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:29,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c226ee4 >2012-12-17 09:19:29,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:29,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4818 >2012-12-17 09:19:29,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4818, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 60e3e146 >2012-12-17 09:19:29,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling started: taskId = b67380cb-25a9-4a7a-abed-a82766cf28aa >2012-12-17 09:19:30,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling ended: taskId = b67380cb-25a9-4a7a-abed-a82766cf28aa task status = finished >2012-12-17 09:19:30,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] spmStart polling ended, spm status: SPM >2012-12-17 09:19:30,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b67380cb-25a9-4a7a-abed-a82766cf28aa), log id: 6ce6e59a >2012-12-17 09:19:30,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMClearTaskVDSCommand, log id: 6ce6e59a >2012-12-17 09:19:30,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2aec1951, log id: 60e3e146 >2012-12-17 09:19:31,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:31,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:31,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bd41e9f >2012-12-17 09:19:31,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bd41e9f >2012-12-17 09:19:31,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c35f8c7 >2012-12-17 09:19:31,222 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:19:31,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63267768 >2012-12-17 09:19:31,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:31,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79e192a4 >2012-12-17 09:19:31,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79e192a4 >2012-12-17 09:19:31,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63267768 >2012-12-17 09:19:31,244 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [4c0a2fb5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:19:32,547 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:19:39,892 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [495d106] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:19:40,029 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [6c899ac8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:19:40,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7acc6f12 >2012-12-17 09:19:40,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 339b65d5 >2012-12-17 09:19:40,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:40,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStopVDSCommand, log id: 339b65d5 >2012-12-17 09:19:40,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, ResetIrsVDSCommand, log id: 7acc6f12 >2012-12-17 09:19:40,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5608f009 >2012-12-17 09:19:40,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:40,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5608f009 >2012-12-17 09:19:42,761 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:19:50,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:50,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4819 >2012-12-17 09:19:50,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4819, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: cad9463 >2012-12-17 09:19:50,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling started: taskId = fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5 >2012-12-17 09:19:51,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended: taskId = fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5 task status = finished >2012-12-17 09:19:51,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] spmStart polling ended, spm status: SPM >2012-12-17 09:19:51,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fc11cb78-5cbe-4ad5-b1e8-afbb833a9ad5), log id: 6325fbe5 >2012-12-17 09:19:51,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, HSMClearTaskVDSCommand, log id: 6325fbe5 >2012-12-17 09:19:51,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [446aaffb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52bcaa3d, log id: cad9463 >2012-12-17 09:19:51,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:51,729 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:19:51,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e078d40 >2012-12-17 09:19:51,745 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [446aaffb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:19:51,925 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [223c8c2d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:19:52,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8d8eebd >2012-12-17 09:19:52,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b25632b >2012-12-17 09:19:52,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:52,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, SpmStopVDSCommand, log id: 6b25632b >2012-12-17 09:19:52,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, ResetIrsVDSCommand, log id: 8d8eebd >2012-12-17 09:19:52,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b9049e >2012-12-17 09:19:52,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:52,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b9049e >2012-12-17 09:19:52,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:19:52,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4820 >2012-12-17 09:19:52,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4820, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54384dfd >2012-12-17 09:19:52,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling started: taskId = 48af515d-803f-4965-a8bb-bb04060012ea >2012-12-17 09:19:53,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling ended: taskId = 48af515d-803f-4965-a8bb-bb04060012ea task status = finished >2012-12-17 09:19:53,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] spmStart polling ended, spm status: SPM >2012-12-17 09:19:53,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48af515d-803f-4965-a8bb-bb04060012ea), log id: f5fa7b5 >2012-12-17 09:19:53,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMClearTaskVDSCommand, log id: f5fa7b5 >2012-12-17 09:19:53,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50fb63f5, log id: 54384dfd >2012-12-17 09:19:53,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:19:53,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:53,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 40b5a4a2 >2012-12-17 09:19:53,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 40b5a4a2 >2012-12-17 09:19:53,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e078d40 >2012-12-17 09:19:53,531 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:19:53,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a3bb291 >2012-12-17 09:19:53,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:19:53,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73dadf5e >2012-12-17 09:19:53,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73dadf5e >2012-12-17 09:19:53,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [25c7cf42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a3bb291 >2012-12-17 09:19:53,551 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [25c7cf42] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Checking autorecoverable hosts >2012-12-17 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Autorecovering 0 hosts >2012-12-17 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Checking autorecoverable hosts done >2012-12-17 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7f190dc3] Autorecovering storage domains is disabled, skipping >2012-12-17 09:20:02,439 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5231d6f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:02,540 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [4f973759] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:02,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 696bd047 >2012-12-17 09:20:02,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68566845 >2012-12-17 09:20:02,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:02,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, SpmStopVDSCommand, log id: 68566845 >2012-12-17 09:20:02,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, ResetIrsVDSCommand, log id: 696bd047 >2012-12-17 09:20:02,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5367f7cb >2012-12-17 09:20:02,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:02,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5367f7cb >2012-12-17 09:20:03,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:20:12,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:12,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4821 >2012-12-17 09:20:12,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4821, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76022377 >2012-12-17 09:20:12,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling started: taskId = 71b16ea4-42ed-4b4f-9ee2-b7c87953add2 >2012-12-17 09:20:13,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended: taskId = 71b16ea4-42ed-4b4f-9ee2-b7c87953add2 task status = finished >2012-12-17 09:20:13,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] spmStart polling ended, spm status: SPM >2012-12-17 09:20:13,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=71b16ea4-42ed-4b4f-9ee2-b7c87953add2), log id: 30d7e649 >2012-12-17 09:20:14,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, HSMClearTaskVDSCommand, log id: 30d7e649 >2012-12-17 09:20:14,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [1d8c274e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e4fdbac, log id: 76022377 >2012-12-17 09:20:14,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:20:14,049 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:20:14,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c06e302 >2012-12-17 09:20:14,080 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [1d8c274e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:14,241 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [e893ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:14,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4ffb02dd >2012-12-17 09:20:14,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b3b9fc1 >2012-12-17 09:20:14,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:14,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStopVDSCommand, log id: b3b9fc1 >2012-12-17 09:20:14,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, ResetIrsVDSCommand, log id: 4ffb02dd >2012-12-17 09:20:14,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 431c5528 >2012-12-17 09:20:14,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:14,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 431c5528 >2012-12-17 09:20:14,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:14,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4822 >2012-12-17 09:20:14,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4822, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23d1642d >2012-12-17 09:20:14,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling started: taskId = e269c584-d23f-4208-b90a-498b15abfe25 >2012-12-17 09:20:15,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended: taskId = e269c584-d23f-4208-b90a-498b15abfe25 task status = finished >2012-12-17 09:20:15,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] spmStart polling ended, spm status: SPM >2012-12-17 09:20:15,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e269c584-d23f-4208-b90a-498b15abfe25), log id: 3ff5e02b >2012-12-17 09:20:15,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMClearTaskVDSCommand, log id: 3ff5e02b >2012-12-17 09:20:15,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@275047db, log id: 23d1642d >2012-12-17 09:20:16,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:20:16,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:16,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 236fbbb4 >2012-12-17 09:20:16,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 236fbbb4 >2012-12-17 09:20:16,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c06e302 >2012-12-17 09:20:16,417 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:20:16,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7c8c3898 >2012-12-17 09:20:16,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:16,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4492fe95 >2012-12-17 09:20:16,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4492fe95 >2012-12-17 09:20:16,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [39739d98] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7c8c3898 >2012-12-17 09:20:16,437 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [39739d98] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:20:23,496 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:20:24,776 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1ebf597f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:24,880 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [3a1c2706] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:24,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2aec6e04 >2012-12-17 09:20:24,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6be35fa3 >2012-12-17 09:20:24,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:25,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStopVDSCommand, log id: 6be35fa3 >2012-12-17 09:20:25,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, ResetIrsVDSCommand, log id: 2aec6e04 >2012-12-17 09:20:25,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 713790b3 >2012-12-17 09:20:25,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:25,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 713790b3 >2012-12-17 09:20:33,738 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:20:35,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:35,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4823 >2012-12-17 09:20:35,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4823, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45df0eb0 >2012-12-17 09:20:35,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling started: taskId = cdcfd788-7350-4642-8159-e4b13ab96a02 >2012-12-17 09:20:36,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling ended: taskId = cdcfd788-7350-4642-8159-e4b13ab96a02 task status = finished >2012-12-17 09:20:36,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] spmStart polling ended, spm status: SPM >2012-12-17 09:20:36,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cdcfd788-7350-4642-8159-e4b13ab96a02), log id: 78b49e9d >2012-12-17 09:20:36,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, HSMClearTaskVDSCommand, log id: 78b49e9d >2012-12-17 09:20:36,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4cfe9a36] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1a199354, log id: 45df0eb0 >2012-12-17 09:20:36,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:20:36,420 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:20:36,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5188e1b3 >2012-12-17 09:20:36,444 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4cfe9a36] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:36,539 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [7f36fad2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:36,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28f6d03e >2012-12-17 09:20:36,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6093ab93 >2012-12-17 09:20:36,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:37,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, SpmStopVDSCommand, log id: 6093ab93 >2012-12-17 09:20:37,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, ResetIrsVDSCommand, log id: 28f6d03e >2012-12-17 09:20:37,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70e1dcf3 >2012-12-17 09:20:37,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7f36fad2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:37,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7f36fad2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70e1dcf3 >2012-12-17 09:20:37,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:37,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4824 >2012-12-17 09:20:37,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4824, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b79778a >2012-12-17 09:20:37,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling started: taskId = e99780cd-1cab-4fd3-8914-f433399592b7 >2012-12-17 09:20:38,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling ended: taskId = e99780cd-1cab-4fd3-8914-f433399592b7 task status = finished >2012-12-17 09:20:38,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] spmStart polling ended, spm status: SPM >2012-12-17 09:20:38,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e99780cd-1cab-4fd3-8914-f433399592b7), log id: 38c43cf5 >2012-12-17 09:20:38,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMClearTaskVDSCommand, log id: 38c43cf5 >2012-12-17 09:20:38,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b7e15f0, log id: 6b79778a >2012-12-17 09:20:38,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:20:38,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:38,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 780e222f >2012-12-17 09:20:38,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 780e222f >2012-12-17 09:20:38,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5188e1b3 >2012-12-17 09:20:38,372 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:20:38,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c21876e >2012-12-17 09:20:38,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:38,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cc4e342 >2012-12-17 09:20:38,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cc4e342 >2012-12-17 09:20:38,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-53) [50bce313] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c21876e >2012-12-17 09:20:38,393 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) [50bce313] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:20:43,934 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:20:47,234 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2ea5eaf2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:47,380 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [2a365c8d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:47,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6175dbc8 >2012-12-17 09:20:47,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1d43046f >2012-12-17 09:20:47,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:47,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, SpmStopVDSCommand, log id: 1d43046f >2012-12-17 09:20:47,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, ResetIrsVDSCommand, log id: 6175dbc8 >2012-12-17 09:20:47,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 397d4c77 >2012-12-17 09:20:47,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:47,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 397d4c77 >2012-12-17 09:20:54,187 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:20:57,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:57,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4825 >2012-12-17 09:20:57,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4825, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4e3714c7 >2012-12-17 09:20:57,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling started: taskId = d6c5ae17-c986-49a5-87d7-cb94893128d7 >2012-12-17 09:20:58,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling ended: taskId = d6c5ae17-c986-49a5-87d7-cb94893128d7 task status = finished >2012-12-17 09:20:58,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] spmStart polling ended, spm status: SPM >2012-12-17 09:20:58,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [30572eed] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d6c5ae17-c986-49a5-87d7-cb94893128d7), log id: 186e2179 >2012-12-17 09:20:58,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, HSMClearTaskVDSCommand, log id: 186e2179 >2012-12-17 09:20:58,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [30572eed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66579b76, log id: 4e3714c7 >2012-12-17 09:20:58,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:20:58,885 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:20:58,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9abcd85 >2012-12-17 09:20:58,914 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [30572eed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:20:59,032 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [f37b622] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:20:59,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7707f73a >2012-12-17 09:20:59,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42ca738 >2012-12-17 09:20:59,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:59,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, SpmStopVDSCommand, log id: 42ca738 >2012-12-17 09:20:59,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, ResetIrsVDSCommand, log id: 7707f73a >2012-12-17 09:20:59,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [f37b622] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76c3f0d5 >2012-12-17 09:20:59,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [f37b622] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:20:59,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [f37b622] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76c3f0d5 >2012-12-17 09:20:59,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:20:59,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4826 >2012-12-17 09:20:59,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4826, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12673c7e >2012-12-17 09:20:59,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling started: taskId = edeed2a1-5b68-48a6-943e-e15a5fb67f00 >2012-12-17 09:21:00,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling ended: taskId = edeed2a1-5b68-48a6-943e-e15a5fb67f00 task status = finished >2012-12-17 09:21:00,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] spmStart polling ended, spm status: SPM >2012-12-17 09:21:00,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=edeed2a1-5b68-48a6-943e-e15a5fb67f00), log id: 628cde93 >2012-12-17 09:21:00,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMClearTaskVDSCommand, log id: 628cde93 >2012-12-17 09:21:00,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e68b771, log id: 12673c7e >2012-12-17 09:21:00,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:21:01,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:01,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 537cf64 >2012-12-17 09:21:01,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 537cf64 >2012-12-17 09:21:01,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9abcd85 >2012-12-17 09:21:01,188 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:01,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1310b0e9 >2012-12-17 09:21:01,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:01,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 41534c19 >2012-12-17 09:21:01,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 41534c19 >2012-12-17 09:21:01,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [67dedb1b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1310b0e9 >2012-12-17 09:21:01,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [67dedb1b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:04,370 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:21:09,401 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [66bf54ed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:21:09,556 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:21:09,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a329a5 >2012-12-17 09:21:09,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 73fa0010 >2012-12-17 09:21:09,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:09,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, SpmStopVDSCommand, log id: 73fa0010 >2012-12-17 09:21:09,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, ResetIrsVDSCommand, log id: 17a329a5 >2012-12-17 09:21:09,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c0cb809 >2012-12-17 09:21:09,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:09,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c0cb809 >2012-12-17 09:21:14,685 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:21:19,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:21:19,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4827 >2012-12-17 09:21:19,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4827, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33431252 >2012-12-17 09:21:19,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling started: taskId = 8db4002a-d19b-40a5-9417-90169acce97c >2012-12-17 09:21:20,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended: taskId = 8db4002a-d19b-40a5-9417-90169acce97c task status = finished >2012-12-17 09:21:20,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] spmStart polling ended, spm status: SPM >2012-12-17 09:21:20,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8db4002a-d19b-40a5-9417-90169acce97c), log id: 7f8953e1 >2012-12-17 09:21:20,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, HSMClearTaskVDSCommand, log id: 7f8953e1 >2012-12-17 09:21:20,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [4dd32292] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10e21a31, log id: 33431252 >2012-12-17 09:21:21,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:21:21,960 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:21:21,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: aa4a356 >2012-12-17 09:21:22,000 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4dd32292] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:21:22,148 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [c8c45fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:21:22,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50d46249 >2012-12-17 09:21:22,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d07bca4 >2012-12-17 09:21:22,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:22,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, SpmStopVDSCommand, log id: 5d07bca4 >2012-12-17 09:21:22,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, ResetIrsVDSCommand, log id: 50d46249 >2012-12-17 09:21:22,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 602665c5 >2012-12-17 09:21:22,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:22,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 602665c5 >2012-12-17 09:21:22,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:21:22,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4828 >2012-12-17 09:21:22,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4828, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7651981b >2012-12-17 09:21:22,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = 25511a55-284e-406f-ae19-8ed081354548 >2012-12-17 09:21:23,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = 25511a55-284e-406f-ae19-8ed081354548 task status = finished >2012-12-17 09:21:23,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM >2012-12-17 09:21:23,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25511a55-284e-406f-ae19-8ed081354548), log id: 7feacef9 >2012-12-17 09:21:23,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 7feacef9 >2012-12-17 09:21:23,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10ba7f0b, log id: 7651981b >2012-12-17 09:21:23,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:21:23,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:23,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bfe10e6 >2012-12-17 09:21:23,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bfe10e6 >2012-12-17 09:21:23,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: aa4a356 >2012-12-17 09:21:23,626 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:23,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6507aba2 >2012-12-17 09:21:23,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:23,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a112126 >2012-12-17 09:21:23,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a112126 >2012-12-17 09:21:23,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6507aba2 >2012-12-17 09:21:23,646 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:24,894 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:21:32,489 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [746c8fa6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:21:32,705 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [13d7977] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:21:32,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5d64de4f >2012-12-17 09:21:32,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a6a04b5 >2012-12-17 09:21:32,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:32,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SpmStopVDSCommand, log id: 4a6a04b5 >2012-12-17 09:21:32,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, ResetIrsVDSCommand, log id: 5d64de4f >2012-12-17 09:21:32,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c7a0cb0 >2012-12-17 09:21:32,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:32,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c7a0cb0 >2012-12-17 09:21:35,094 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:21:43,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:21:43,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4829 >2012-12-17 09:21:43,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4829, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e7895dd >2012-12-17 09:21:43,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling started: taskId = 8e733400-8413-40ae-a18d-3c99f65dc527 >2012-12-17 09:21:44,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling ended: taskId = 8e733400-8413-40ae-a18d-3c99f65dc527 task status = finished >2012-12-17 09:21:44,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] spmStart polling ended, spm status: SPM >2012-12-17 09:21:44,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8e733400-8413-40ae-a18d-3c99f65dc527), log id: 29233a1f >2012-12-17 09:21:44,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, HSMClearTaskVDSCommand, log id: 29233a1f >2012-12-17 09:21:44,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2dc4111d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47bcde52, log id: 2e7895dd >2012-12-17 09:21:44,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:21:44,303 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:21:44,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 479454a >2012-12-17 09:21:44,343 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2dc4111d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:21:44,488 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [1ed39251] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:21:44,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68693dad >2012-12-17 09:21:44,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25a81e0b >2012-12-17 09:21:44,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:44,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, SpmStopVDSCommand, log id: 25a81e0b >2012-12-17 09:21:44,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, ResetIrsVDSCommand, log id: 68693dad >2012-12-17 09:21:44,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4046b736 >2012-12-17 09:21:44,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:44,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [1ed39251] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4046b736 >2012-12-17 09:21:44,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:21:44,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4830 >2012-12-17 09:21:44,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4830, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: da0e83d >2012-12-17 09:21:44,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07 >2012-12-17 09:21:46,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07 task status = finished >2012-12-17 09:21:46,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM >2012-12-17 09:21:46,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f279b8bc-cb3d-40cc-b0b8-bcc2d22e5c07), log id: 4f379825 >2012-12-17 09:21:46,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 4f379825 >2012-12-17 09:21:46,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ae5cdc4, log id: da0e83d >2012-12-17 09:21:46,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:21:46,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:46,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d07578 >2012-12-17 09:21:46,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d07578 >2012-12-17 09:21:46,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 479454a >2012-12-17 09:21:46,558 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:46,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 38d01c43 >2012-12-17 09:21:46,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:46,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24a9b363 >2012-12-17 09:21:46,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24a9b363 >2012-12-17 09:21:46,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 38d01c43 >2012-12-17 09:21:46,578 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:21:55,011 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [757df811] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:21:55,134 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [160c510e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:21:55,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ca2688e >2012-12-17 09:21:55,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3b8717c1 >2012-12-17 09:21:55,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:55,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStopVDSCommand, log id: 3b8717c1 >2012-12-17 09:21:55,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, ResetIrsVDSCommand, log id: 5ca2688e >2012-12-17 09:21:55,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 725f5a0f >2012-12-17 09:21:55,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:21:55,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 725f5a0f >2012-12-17 09:21:56,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:22:05,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:06,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4831 >2012-12-17 09:22:06,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4831, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 286c40fa >2012-12-17 09:22:06,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling started: taskId = 07f607b4-8398-4ddf-af11-27aba8b9e7bc >2012-12-17 09:22:07,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling ended: taskId = 07f607b4-8398-4ddf-af11-27aba8b9e7bc task status = finished >2012-12-17 09:22:07,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] spmStart polling ended, spm status: SPM >2012-12-17 09:22:07,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=07f607b4-8398-4ddf-af11-27aba8b9e7bc), log id: 5023a60c >2012-12-17 09:22:07,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, HSMClearTaskVDSCommand, log id: 5023a60c >2012-12-17 09:22:07,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@546a86ee, log id: 286c40fa >2012-12-17 09:22:07,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:07,539 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:22:07,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f6a4983 >2012-12-17 09:22:07,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1ca2c4d4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:22:07,714 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [29b31be1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:22:07,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20b9c0f3 >2012-12-17 09:22:07,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 604a0eb1 >2012-12-17 09:22:07,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:08,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SpmStopVDSCommand, log id: 604a0eb1 >2012-12-17 09:22:08,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, ResetIrsVDSCommand, log id: 20b9c0f3 >2012-12-17 09:22:08,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50a4f8c9 >2012-12-17 09:22:08,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:08,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50a4f8c9 >2012-12-17 09:22:08,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:08,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4832 >2012-12-17 09:22:08,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4832, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59ad3443 >2012-12-17 09:22:08,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling started: taskId = 6b5141df-3a54-4ceb-a464-b18fffd8753a >2012-12-17 09:22:09,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended: taskId = 6b5141df-3a54-4ceb-a464-b18fffd8753a task status = finished >2012-12-17 09:22:09,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended, spm status: SPM >2012-12-17 09:22:09,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6b5141df-3a54-4ceb-a464-b18fffd8753a), log id: 20441ed0 >2012-12-17 09:22:09,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMClearTaskVDSCommand, log id: 20441ed0 >2012-12-17 09:22:09,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5842fa3c, log id: 59ad3443 >2012-12-17 09:22:09,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:09,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:09,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 442f0660 >2012-12-17 09:22:09,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 442f0660 >2012-12-17 09:22:09,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f6a4983 >2012-12-17 09:22:09,349 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:09,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b4bab98 >2012-12-17 09:22:09,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:09,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bcb3a >2012-12-17 09:22:09,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bcb3a >2012-12-17 09:22:09,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b4bab98 >2012-12-17 09:22:09,369 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [3299bae4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:17,174 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:22:18,236 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [25c7cf42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:22:18,556 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [5d07077e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:22:18,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3a75e4d7 >2012-12-17 09:22:18,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ed0c979 >2012-12-17 09:22:18,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:18,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, SpmStopVDSCommand, log id: 7ed0c979 >2012-12-17 09:22:18,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, ResetIrsVDSCommand, log id: 3a75e4d7 >2012-12-17 09:22:18,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74fb38c7 >2012-12-17 09:22:18,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:18,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74fb38c7 >2012-12-17 09:22:27,350 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:22:29,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:29,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4833 >2012-12-17 09:22:29,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4833, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 656f51fe >2012-12-17 09:22:29,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling started: taskId = c2b83930-508a-4770-88da-6a8e6a4dc82b >2012-12-17 09:22:30,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling ended: taskId = c2b83930-508a-4770-88da-6a8e6a4dc82b task status = finished >2012-12-17 09:22:30,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] spmStart polling ended, spm status: SPM >2012-12-17 09:22:30,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c2b83930-508a-4770-88da-6a8e6a4dc82b), log id: 60009592 >2012-12-17 09:22:30,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, HSMClearTaskVDSCommand, log id: 60009592 >2012-12-17 09:22:30,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [1513ca5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@465bd57b, log id: 656f51fe >2012-12-17 09:22:30,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:30,241 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:22:30,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59d58215 >2012-12-17 09:22:30,278 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [1513ca5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:22:31,629 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [26d9e89a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:22:31,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 507a42a2 >2012-12-17 09:22:31,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 55ad106c >2012-12-17 09:22:31,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:31,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStopVDSCommand, log id: 55ad106c >2012-12-17 09:22:31,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, ResetIrsVDSCommand, log id: 507a42a2 >2012-12-17 09:22:31,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79486898 >2012-12-17 09:22:31,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:31,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79486898 >2012-12-17 09:22:32,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:32,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4834 >2012-12-17 09:22:32,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4834, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 259060ea >2012-12-17 09:22:32,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling started: taskId = 63805827-a23d-46f9-aacf-fa9b638a4fde >2012-12-17 09:22:33,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended: taskId = 63805827-a23d-46f9-aacf-fa9b638a4fde task status = finished >2012-12-17 09:22:33,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended, spm status: SPM >2012-12-17 09:22:33,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=63805827-a23d-46f9-aacf-fa9b638a4fde), log id: 2c80b98e >2012-12-17 09:22:33,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMClearTaskVDSCommand, log id: 2c80b98e >2012-12-17 09:22:33,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37163075, log id: 259060ea >2012-12-17 09:22:33,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:33,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:33,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 48b366c1 >2012-12-17 09:22:33,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48b366c1 >2012-12-17 09:22:33,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59d58215 >2012-12-17 09:22:33,257 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:33,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15118ff3 >2012-12-17 09:22:33,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:33,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 629c5c2a >2012-12-17 09:22:33,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 629c5c2a >2012-12-17 09:22:33,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15118ff3 >2012-12-17 09:22:33,277 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [779086b5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:37,725 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:22:42,096 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4c0a2fb5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:22:42,240 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [15f75f8d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:22:42,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23377118 >2012-12-17 09:22:42,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e4ce48 >2012-12-17 09:22:42,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:43,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, SpmStopVDSCommand, log id: 7e4ce48 >2012-12-17 09:22:43,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, ResetIrsVDSCommand, log id: 23377118 >2012-12-17 09:22:43,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6394bb05 >2012-12-17 09:22:43,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:43,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [15f75f8d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6394bb05 >2012-12-17 09:22:47,905 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:22:53,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:53,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4835 >2012-12-17 09:22:53,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4835, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799ad51d >2012-12-17 09:22:53,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling started: taskId = 2fe3b427-5fa5-4326-9b21-ba1054903ea1 >2012-12-17 09:22:54,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended: taskId = 2fe3b427-5fa5-4326-9b21-ba1054903ea1 task status = finished >2012-12-17 09:22:54,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] spmStart polling ended, spm status: SPM >2012-12-17 09:22:54,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2fe3b427-5fa5-4326-9b21-ba1054903ea1), log id: 37da5da4 >2012-12-17 09:22:54,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, HSMClearTaskVDSCommand, log id: 37da5da4 >2012-12-17 09:22:54,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [3e0f12eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6139d6a1, log id: 799ad51d >2012-12-17 09:22:54,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:54,475 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:22:54,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44211788 >2012-12-17 09:22:54,517 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e0f12eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:22:54,686 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [4f3078ed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:22:54,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bed9894 >2012-12-17 09:22:54,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c277eb0 >2012-12-17 09:22:54,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:55,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SpmStopVDSCommand, log id: 6c277eb0 >2012-12-17 09:22:55,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, ResetIrsVDSCommand, log id: 2bed9894 >2012-12-17 09:22:55,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b45e087 >2012-12-17 09:22:55,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:55,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b45e087 >2012-12-17 09:22:55,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:22:55,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4836 >2012-12-17 09:22:55,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4836, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76474bae >2012-12-17 09:22:55,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling started: taskId = 24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd >2012-12-17 09:22:56,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling ended: taskId = 24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd task status = finished >2012-12-17 09:22:56,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] spmStart polling ended, spm status: SPM >2012-12-17 09:22:56,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=24c8b5cf-a9ff-4c3c-a25b-40d9bd9593cd), log id: 2761c1c >2012-12-17 09:22:56,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMClearTaskVDSCommand, log id: 2761c1c >2012-12-17 09:22:56,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@182f7387, log id: 76474bae >2012-12-17 09:22:56,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:22:56,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:56,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 685b56dd >2012-12-17 09:22:56,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 685b56dd >2012-12-17 09:22:56,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44211788 >2012-12-17 09:22:56,337 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:56,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44a3fcb1 >2012-12-17 09:22:56,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:22:56,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c79b091 >2012-12-17 09:22:56,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c79b091 >2012-12-17 09:22:56,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-9) [29b31be1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44a3fcb1 >2012-12-17 09:22:56,357 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) [29b31be1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:22:58,120 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:23:05,284 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [11c9636b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:23:05,521 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [5389729] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:23:06,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 38c7f479 >2012-12-17 09:23:06,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 25aad06f >2012-12-17 09:23:06,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:06,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SpmStopVDSCommand, log id: 25aad06f >2012-12-17 09:23:06,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, ResetIrsVDSCommand, log id: 38c7f479 >2012-12-17 09:23:06,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d60f920 >2012-12-17 09:23:06,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:06,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d60f920 >2012-12-17 09:23:08,310 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:23:16,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:23:16,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4837 >2012-12-17 09:23:16,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4837, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a4ad157 >2012-12-17 09:23:16,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling started: taskId = 19c20a09-2566-4482-9e7b-bdb4ad0d84db >2012-12-17 09:23:17,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended: taskId = 19c20a09-2566-4482-9e7b-bdb4ad0d84db task status = finished >2012-12-17 09:23:17,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] spmStart polling ended, spm status: SPM >2012-12-17 09:23:17,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19c20a09-2566-4482-9e7b-bdb4ad0d84db), log id: 687b0caa >2012-12-17 09:23:17,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, HSMClearTaskVDSCommand, log id: 687b0caa >2012-12-17 09:23:17,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [63fee526] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@667ca27c, log id: 4a4ad157 >2012-12-17 09:23:17,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:23:17,648 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:23:17,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3118d901 >2012-12-17 09:23:17,665 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [63fee526] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:23:17,799 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [10684b21] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:23:17,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7e73b609 >2012-12-17 09:23:17,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f4ff669 >2012-12-17 09:23:17,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:18,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, SpmStopVDSCommand, log id: 2f4ff669 >2012-12-17 09:23:18,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, ResetIrsVDSCommand, log id: 7e73b609 >2012-12-17 09:23:18,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [10684b21] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c519eb8 >2012-12-17 09:23:18,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [10684b21] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:18,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [10684b21] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c519eb8 >2012-12-17 09:23:18,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:23:18,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4838 >2012-12-17 09:23:18,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4838, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40a9aa9b >2012-12-17 09:23:18,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling started: taskId = c9a10d00-e1e1-4748-a6d8-a5e551beae5b >2012-12-17 09:23:19,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended: taskId = c9a10d00-e1e1-4748-a6d8-a5e551beae5b task status = finished >2012-12-17 09:23:19,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended, spm status: SPM >2012-12-17 09:23:19,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c9a10d00-e1e1-4748-a6d8-a5e551beae5b), log id: 5fd28ca5 >2012-12-17 09:23:19,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMClearTaskVDSCommand, log id: 5fd28ca5 >2012-12-17 09:23:19,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@780fd5c8, log id: 40a9aa9b >2012-12-17 09:23:19,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:23:19,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:19,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 30ba2e28 >2012-12-17 09:23:19,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 30ba2e28 >2012-12-17 09:23:19,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3118d901 >2012-12-17 09:23:19,279 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:23:19,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f2c5d16 >2012-12-17 09:23:19,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:19,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f357b97 >2012-12-17 09:23:19,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f357b97 >2012-12-17 09:23:19,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f2c5d16 >2012-12-17 09:23:19,300 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [57cdd675] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:23:28,150 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [39739d98] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:23:28,272 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [492c9849] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:23:28,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5e30530d >2012-12-17 09:23:28,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2f679194 >2012-12-17 09:23:28,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:28,804 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:23:28,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStopVDSCommand, log id: 2f679194 >2012-12-17 09:23:28,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, ResetIrsVDSCommand, log id: 5e30530d >2012-12-17 09:23:28,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34e729dc >2012-12-17 09:23:28,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:28,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34e729dc >2012-12-17 09:23:39,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:23:39,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4839 >2012-12-17 09:23:39,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4839, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4597a118 >2012-12-17 09:23:39,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling started: taskId = cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83 >2012-12-17 09:23:40,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended: taskId = cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83 task status = finished >2012-12-17 09:23:40,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] spmStart polling ended, spm status: SPM >2012-12-17 09:23:40,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cf1fcb5a-1d7e-45cc-9b0e-98f882d2ef83), log id: 603f2b95 >2012-12-17 09:23:40,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, HSMClearTaskVDSCommand, log id: 603f2b95 >2012-12-17 09:23:40,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [441eec2e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53467490, log id: 4597a118 >2012-12-17 09:23:40,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:23:40,248 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:23:40,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68005028 >2012-12-17 09:23:40,257 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [441eec2e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:23:41,019 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [8d9c42c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:23:41,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 64418f3 >2012-12-17 09:23:41,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d83a734 >2012-12-17 09:23:41,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:41,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, SpmStopVDSCommand, log id: d83a734 >2012-12-17 09:23:41,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, ResetIrsVDSCommand, log id: 64418f3 >2012-12-17 09:23:41,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a37099d >2012-12-17 09:23:41,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [8d9c42c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:41,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [8d9c42c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a37099d >2012-12-17 09:23:41,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:23:41,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4840 >2012-12-17 09:23:41,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4840, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7410820 >2012-12-17 09:23:41,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8 >2012-12-17 09:23:42,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8 task status = finished >2012-12-17 09:23:42,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM >2012-12-17 09:23:42,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b3c0c5c0-85b1-40b8-b4f1-b83ba5afe9d8), log id: 74f49533 >2012-12-17 09:23:42,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 74f49533 >2012-12-17 09:23:42,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@656f8341, log id: 7410820 >2012-12-17 09:23:42,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:23:42,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:42,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7a6ba0bf >2012-12-17 09:23:43,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a6ba0bf >2012-12-17 09:23:43,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68005028 >2012-12-17 09:23:43,016 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:23:43,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31eb6dce >2012-12-17 09:23:43,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:43,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 21316151 >2012-12-17 09:23:43,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 21316151 >2012-12-17 09:23:43,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31eb6dce >2012-12-17 09:23:43,043 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:23:49,235 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:23:51,908 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1091267a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:23:52,104 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:23:52,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 73d89590 >2012-12-17 09:23:52,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b4f0d60 >2012-12-17 09:23:52,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:52,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, SpmStopVDSCommand, log id: 4b4f0d60 >2012-12-17 09:23:52,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, ResetIrsVDSCommand, log id: 73d89590 >2012-12-17 09:23:52,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44746e7e >2012-12-17 09:23:52,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:23:52,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44746e7e >2012-12-17 09:23:59,549 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:02,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:02,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4841 >2012-12-17 09:24:02,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4841, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c45670 >2012-12-17 09:24:02,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling started: taskId = 965be689-3e54-4f45-aed3-ede1fafe8c34 >2012-12-17 09:24:03,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling ended: taskId = 965be689-3e54-4f45-aed3-ede1fafe8c34 task status = finished >2012-12-17 09:24:03,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] spmStart polling ended, spm status: SPM >2012-12-17 09:24:03,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [4f973759] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=965be689-3e54-4f45-aed3-ede1fafe8c34), log id: 317cf53a >2012-12-17 09:24:03,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, HSMClearTaskVDSCommand, log id: 317cf53a >2012-12-17 09:24:03,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [4f973759] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c7cc38, log id: 6c45670 >2012-12-17 09:24:03,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:03,800 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:24:03,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 503c8411 >2012-12-17 09:24:03,839 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4f973759] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:03,974 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [2bb64a93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:04,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10aaa9d8 >2012-12-17 09:24:04,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78acda63 >2012-12-17 09:24:04,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:04,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, SpmStopVDSCommand, log id: 78acda63 >2012-12-17 09:24:04,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, ResetIrsVDSCommand, log id: 10aaa9d8 >2012-12-17 09:24:04,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57f60efd >2012-12-17 09:24:04,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:04,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2bb64a93] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57f60efd >2012-12-17 09:24:04,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:04,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4842 >2012-12-17 09:24:04,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4842, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b2a2d4c >2012-12-17 09:24:04,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = ac189118-8df0-4262-aefe-ff216783a77a >2012-12-17 09:24:05,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = ac189118-8df0-4262-aefe-ff216783a77a task status = finished >2012-12-17 09:24:05,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM >2012-12-17 09:24:05,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ac189118-8df0-4262-aefe-ff216783a77a), log id: 3d51b29c >2012-12-17 09:24:05,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 3d51b29c >2012-12-17 09:24:05,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ea53f37, log id: 4b2a2d4c >2012-12-17 09:24:06,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:06,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:06,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d5a967 >2012-12-17 09:24:06,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d5a967 >2012-12-17 09:24:06,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 503c8411 >2012-12-17 09:24:06,245 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:06,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f4f36ee >2012-12-17 09:24:06,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:06,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d9b2939 >2012-12-17 09:24:06,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d9b2939 >2012-12-17 09:24:06,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f4f36ee >2012-12-17 09:24:06,265 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:09,783 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:14,550 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [14ba2ea4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:14,694 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [127bfc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:14,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c0fe961 >2012-12-17 09:24:14,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 251a3248 >2012-12-17 09:24:14,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:15,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStopVDSCommand, log id: 251a3248 >2012-12-17 09:24:15,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, ResetIrsVDSCommand, log id: 6c0fe961 >2012-12-17 09:24:15,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d573d2d >2012-12-17 09:24:15,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:15,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d573d2d >2012-12-17 09:24:20,014 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:25,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:25,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4843 >2012-12-17 09:24:25,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4843, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6af5d471 >2012-12-17 09:24:25,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling started: taskId = a3375965-d815-4a08-abd7-6b140099c88f >2012-12-17 09:24:26,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended: taskId = a3375965-d815-4a08-abd7-6b140099c88f task status = finished >2012-12-17 09:24:26,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] spmStart polling ended, spm status: SPM >2012-12-17 09:24:26,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3375965-d815-4a08-abd7-6b140099c88f), log id: 71a735df >2012-12-17 09:24:26,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, HSMClearTaskVDSCommand, log id: 71a735df >2012-12-17 09:24:26,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [1598e29f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18c1fdc4, log id: 6af5d471 >2012-12-17 09:24:26,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:26,827 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:24:26,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5457e27a >2012-12-17 09:24:26,857 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1598e29f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:27,066 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [6b59f114] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:27,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5852e09d >2012-12-17 09:24:27,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 170a08a4 >2012-12-17 09:24:27,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:27,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, SpmStopVDSCommand, log id: 170a08a4 >2012-12-17 09:24:27,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, ResetIrsVDSCommand, log id: 5852e09d >2012-12-17 09:24:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f6ca323 >2012-12-17 09:24:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f6ca323 >2012-12-17 09:24:27,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:27,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4844 >2012-12-17 09:24:27,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4844, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 375803e1 >2012-12-17 09:24:27,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling started: taskId = 7c4d6ffd-9b5d-473c-b660-d69564859127 >2012-12-17 09:24:28,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended: taskId = 7c4d6ffd-9b5d-473c-b660-d69564859127 task status = finished >2012-12-17 09:24:28,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended, spm status: SPM >2012-12-17 09:24:28,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7c4d6ffd-9b5d-473c-b660-d69564859127), log id: 70f48923 >2012-12-17 09:24:28,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMClearTaskVDSCommand, log id: 70f48923 >2012-12-17 09:24:28,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@465792c4, log id: 375803e1 >2012-12-17 09:24:28,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:28,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:28,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9b9f510 >2012-12-17 09:24:28,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9b9f510 >2012-12-17 09:24:28,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5457e27a >2012-12-17 09:24:28,539 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:28,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f87f26d >2012-12-17 09:24:28,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:28,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 49ac91b >2012-12-17 09:24:28,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 49ac91b >2012-12-17 09:24:28,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f87f26d >2012-12-17 09:24:28,560 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:30,266 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:37,416 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [50bce313] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:37,610 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [78e614fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:37,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6520d334 >2012-12-17 09:24:37,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2edbe137 >2012-12-17 09:24:37,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:37,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, SpmStopVDSCommand, log id: 2edbe137 >2012-12-17 09:24:37,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, ResetIrsVDSCommand, log id: 6520d334 >2012-12-17 09:24:37,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72065448 >2012-12-17 09:24:37,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:37,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [78e614fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72065448 >2012-12-17 09:24:41,640 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:47,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:47,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4845 >2012-12-17 09:24:47,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4845, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7868539b >2012-12-17 09:24:47,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling started: taskId = 3bea3675-b1b2-435c-b66b-848e8cfcbde2 >2012-12-17 09:24:48,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling ended: taskId = 3bea3675-b1b2-435c-b66b-848e8cfcbde2 task status = finished >2012-12-17 09:24:48,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] spmStart polling ended, spm status: SPM >2012-12-17 09:24:48,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3bea3675-b1b2-435c-b66b-848e8cfcbde2), log id: 7015ce46 >2012-12-17 09:24:49,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, HSMClearTaskVDSCommand, log id: 7015ce46 >2012-12-17 09:24:49,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [356a6c1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27998f27, log id: 7868539b >2012-12-17 09:24:49,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:49,075 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:24:49,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e52ba00 >2012-12-17 09:24:49,101 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [356a6c1f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:49,290 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [3e7d3817] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:49,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13636d9a >2012-12-17 09:24:49,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3edd8e16 >2012-12-17 09:24:49,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:49,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStopVDSCommand, log id: 3edd8e16 >2012-12-17 09:24:49,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, ResetIrsVDSCommand, log id: 13636d9a >2012-12-17 09:24:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11d625e0 >2012-12-17 09:24:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11d625e0 >2012-12-17 09:24:49,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:24:49,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4846 >2012-12-17 09:24:49,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4846, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bf83726 >2012-12-17 09:24:49,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling started: taskId = 0f536468-8aea-4ef0-ad0c-49717ee13738 >2012-12-17 09:24:50,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended: taskId = 0f536468-8aea-4ef0-ad0c-49717ee13738 task status = finished >2012-12-17 09:24:50,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] spmStart polling ended, spm status: SPM >2012-12-17 09:24:50,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0f536468-8aea-4ef0-ad0c-49717ee13738), log id: 40248a40 >2012-12-17 09:24:50,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMClearTaskVDSCommand, log id: 40248a40 >2012-12-17 09:24:50,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ef090a6, log id: 2bf83726 >2012-12-17 09:24:51,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:24:51,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:51,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 17b1caa3 >2012-12-17 09:24:51,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 17b1caa3 >2012-12-17 09:24:51,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e52ba00 >2012-12-17 09:24:51,276 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:51,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41f61374 >2012-12-17 09:24:51,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:24:51,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 47f95728 >2012-12-17 09:24:51,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47f95728 >2012-12-17 09:24:51,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [33cde09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41f61374 >2012-12-17 09:24:51,298 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [33cde09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:24:51,861 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:24:59,691 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2ed19e27] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:24:59,834 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [3c6804c8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:24:59,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 295a0ab5 >2012-12-17 09:24:59,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10c268a6 >2012-12-17 09:24:59,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Checking autorecoverable hosts >2012-12-17 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Autorecovering 0 hosts >2012-12-17 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Checking autorecoverable hosts done >2012-12-17 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [61136831] Autorecovering storage domains is disabled, skipping >2012-12-17 09:25:00,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, SpmStopVDSCommand, log id: 10c268a6 >2012-12-17 09:25:00,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, ResetIrsVDSCommand, log id: 295a0ab5 >2012-12-17 09:25:00,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b4970d4 >2012-12-17 09:25:00,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:00,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b4970d4 >2012-12-17 09:25:02,124 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:25:10,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:10,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4847 >2012-12-17 09:25:10,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4847, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8a1114d >2012-12-17 09:25:10,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling started: taskId = 83f45c58-a8a4-4ea5-b2e2-c236d58d1b28 >2012-12-17 09:25:11,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended: taskId = 83f45c58-a8a4-4ea5-b2e2-c236d58d1b28 task status = finished >2012-12-17 09:25:11,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] spmStart polling ended, spm status: SPM >2012-12-17 09:25:11,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=83f45c58-a8a4-4ea5-b2e2-c236d58d1b28), log id: 2b25f467 >2012-12-17 09:25:11,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, HSMClearTaskVDSCommand, log id: 2b25f467 >2012-12-17 09:25:11,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [1756fb88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48d0ac52, log id: 8a1114d >2012-12-17 09:25:11,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:25:11,663 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:25:11,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3644438c >2012-12-17 09:25:11,748 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [1756fb88] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:25:12,042 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [78864454] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:25:12,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61f23c49 >2012-12-17 09:25:12,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 47bd7efc >2012-12-17 09:25:12,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:12,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, SpmStopVDSCommand, log id: 47bd7efc >2012-12-17 09:25:12,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, ResetIrsVDSCommand, log id: 61f23c49 >2012-12-17 09:25:12,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [78864454] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c6ef5a7 >2012-12-17 09:25:12,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:12,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [78864454] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c6ef5a7 >2012-12-17 09:25:12,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:12,421 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:25:12,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4848 >2012-12-17 09:25:12,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4848, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ad66dd3 >2012-12-17 09:25:12,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling started: taskId = 9cbcf23f-acc2-43ac-80d3-ae62ba945a39 >2012-12-17 09:25:13,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended: taskId = 9cbcf23f-acc2-43ac-80d3-ae62ba945a39 task status = finished >2012-12-17 09:25:13,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended, spm status: SPM >2012-12-17 09:25:13,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9cbcf23f-acc2-43ac-80d3-ae62ba945a39), log id: 7fb909f4 >2012-12-17 09:25:13,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMClearTaskVDSCommand, log id: 7fb909f4 >2012-12-17 09:25:13,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d3cb3ae, log id: 3ad66dd3 >2012-12-17 09:25:13,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:25:13,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:13,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29382d2e >2012-12-17 09:25:13,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29382d2e >2012-12-17 09:25:13,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3644438c >2012-12-17 09:25:13,674 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:13,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 271e3c26 >2012-12-17 09:25:13,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:13,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 684c0f1f >2012-12-17 09:25:13,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 684c0f1f >2012-12-17 09:25:13,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 271e3c26 >2012-12-17 09:25:13,697 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:13,738 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:25:13,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b130c3a >2012-12-17 09:25:13,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:13,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d9645da >2012-12-17 09:25:13,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d9645da >2012-12-17 09:25:13,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b130c3a >2012-12-17 09:25:13,765 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:13,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 12f7cfdc >2012-12-17 09:25:13,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:13,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4beaf561 >2012-12-17 09:25:13,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4beaf561 >2012-12-17 09:25:13,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 12f7cfdc >2012-12-17 09:25:13,786 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [69834199] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:22,506 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7f36fad2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:25:22,828 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [550a1bf7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:25:22,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47912c94 >2012-12-17 09:25:22,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 54847281 >2012-12-17 09:25:22,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:23,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, SpmStopVDSCommand, log id: 54847281 >2012-12-17 09:25:23,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, ResetIrsVDSCommand, log id: 47912c94 >2012-12-17 09:25:23,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e1a5c65 >2012-12-17 09:25:23,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:23,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e1a5c65 >2012-12-17 09:25:33,021 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:25:33,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:33,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4849 >2012-12-17 09:25:33,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4849, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 625bcc0b >2012-12-17 09:25:33,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling started: taskId = f0f48372-f3f9-455f-ad5d-c493d5d36334 >2012-12-17 09:25:34,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling ended: taskId = f0f48372-f3f9-455f-ad5d-c493d5d36334 task status = finished >2012-12-17 09:25:34,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] spmStart polling ended, spm status: SPM >2012-12-17 09:25:34,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f0f48372-f3f9-455f-ad5d-c493d5d36334), log id: abd074d >2012-12-17 09:25:34,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, HSMClearTaskVDSCommand, log id: abd074d >2012-12-17 09:25:34,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [3eeaa12a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a55bd, log id: 625bcc0b >2012-12-17 09:25:34,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:25:34,972 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:25:34,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c314c8d >2012-12-17 09:25:34,984 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [3eeaa12a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:25:35,220 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [36643dd7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:25:35,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dd4f0b2 >2012-12-17 09:25:35,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4286f156 >2012-12-17 09:25:35,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:35,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStopVDSCommand, log id: 4286f156 >2012-12-17 09:25:36,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, ResetIrsVDSCommand, log id: dd4f0b2 >2012-12-17 09:25:36,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59118b5d >2012-12-17 09:25:36,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:36,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59118b5d >2012-12-17 09:25:36,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:36,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4850 >2012-12-17 09:25:36,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4850, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1237ccaf >2012-12-17 09:25:36,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling started: taskId = 750daa0b-3f7e-4c8b-add1-80b61487b11e >2012-12-17 09:25:37,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended: taskId = 750daa0b-3f7e-4c8b-add1-80b61487b11e task status = finished >2012-12-17 09:25:37,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] spmStart polling ended, spm status: SPM >2012-12-17 09:25:37,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=750daa0b-3f7e-4c8b-add1-80b61487b11e), log id: 564a58d0 >2012-12-17 09:25:37,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMClearTaskVDSCommand, log id: 564a58d0 >2012-12-17 09:25:37,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b0cebf2, log id: 1237ccaf >2012-12-17 09:25:37,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:25:37,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:37,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4fbda1de >2012-12-17 09:25:37,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4fbda1de >2012-12-17 09:25:37,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c314c8d >2012-12-17 09:25:37,390 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:37,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2ace1eb8 >2012-12-17 09:25:37,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:37,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 58e0114b >2012-12-17 09:25:37,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 58e0114b >2012-12-17 09:25:37,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [a029edc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2ace1eb8 >2012-12-17 09:25:37,410 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [a029edc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:25:43,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:25:46,446 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [65048227] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:25:47,003 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [622cf65b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:25:47,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: dd0ecee >2012-12-17 09:25:47,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4d96372c >2012-12-17 09:25:47,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:47,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, SpmStopVDSCommand, log id: 4d96372c >2012-12-17 09:25:47,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, ResetIrsVDSCommand, log id: dd0ecee >2012-12-17 09:25:47,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e8eb342 >2012-12-17 09:25:47,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:47,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [622cf65b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e8eb342 >2012-12-17 09:25:51,777 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) [76c72cb4] DbUserCacheManager::refreshAllUserData() - entered >2012-12-17 09:25:53,450 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:25:57,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:57,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4851 >2012-12-17 09:25:57,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4851, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b42984e >2012-12-17 09:25:57,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling started: taskId = d6c93898-85dc-415f-b4a6-d3f58dfcd9e3 >2012-12-17 09:25:58,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling ended: taskId = d6c93898-85dc-415f-b4a6-d3f58dfcd9e3 task status = finished >2012-12-17 09:25:58,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] spmStart polling ended, spm status: SPM >2012-12-17 09:25:58,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d6c93898-85dc-415f-b4a6-d3f58dfcd9e3), log id: 1d2ad14e >2012-12-17 09:25:58,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, HSMClearTaskVDSCommand, log id: 1d2ad14e >2012-12-17 09:25:58,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [6c5669f4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@661366d7, log id: 4b42984e >2012-12-17 09:25:58,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:25:58,711 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:25:58,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39edc935 >2012-12-17 09:25:58,743 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [6c5669f4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:25:58,939 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [45dceb6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:25:59,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fea0990 >2012-12-17 09:25:59,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4cdd48d9 >2012-12-17 09:25:59,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:59,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, SpmStopVDSCommand, log id: 4cdd48d9 >2012-12-17 09:25:59,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, ResetIrsVDSCommand, log id: 6fea0990 >2012-12-17 09:25:59,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 638590c3 >2012-12-17 09:25:59,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [45dceb6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:25:59,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [45dceb6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 638590c3 >2012-12-17 09:25:59,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:25:59,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4852 >2012-12-17 09:25:59,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4852, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40f8571e >2012-12-17 09:25:59,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = fbef59f5-1b2b-4197-82b0-f38ee1590453 >2012-12-17 09:26:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = fbef59f5-1b2b-4197-82b0-f38ee1590453 task status = finished >2012-12-17 09:26:00,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 09:26:00,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=fbef59f5-1b2b-4197-82b0-f38ee1590453), log id: 3d54a41b >2012-12-17 09:26:00,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 3d54a41b >2012-12-17 09:26:00,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7fe20811, log id: 40f8571e >2012-12-17 09:26:00,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:26:00,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d8c3f01 >2012-12-17 09:26:00,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d8c3f01 >2012-12-17 09:26:00,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39edc935 >2012-12-17 09:26:00,414 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:00,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 127c92d7 >2012-12-17 09:26:00,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:00,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7f740b86 >2012-12-17 09:26:00,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f740b86 >2012-12-17 09:26:00,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 127c92d7 >2012-12-17 09:26:00,435 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-88) [5c1c47fa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:03,704 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:26:09,304 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4beddd99] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:26:09,428 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [7bb72f96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:26:09,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 37b2c114 >2012-12-17 09:26:09,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7ebbf570 >2012-12-17 09:26:09,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:09,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, SpmStopVDSCommand, log id: 7ebbf570 >2012-12-17 09:26:09,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, ResetIrsVDSCommand, log id: 37b2c114 >2012-12-17 09:26:09,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76d493e6 >2012-12-17 09:26:09,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [7bb72f96] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:09,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [7bb72f96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76d493e6 >2012-12-17 09:26:13,893 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:26:19,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:26:19,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4853 >2012-12-17 09:26:19,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4853, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 613ebde8 >2012-12-17 09:26:19,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling started: taskId = 765e2c65-6271-4a40-ac27-e53609941adb >2012-12-17 09:26:20,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling ended: taskId = 765e2c65-6271-4a40-ac27-e53609941adb task status = finished >2012-12-17 09:26:20,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] spmStart polling ended, spm status: SPM >2012-12-17 09:26:20,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=765e2c65-6271-4a40-ac27-e53609941adb), log id: 38d03ac7 >2012-12-17 09:26:20,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, HSMClearTaskVDSCommand, log id: 38d03ac7 >2012-12-17 09:26:20,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [5ecdfb51] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ec2fbf4, log id: 613ebde8 >2012-12-17 09:26:21,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:26:21,926 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:26:21,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b7532ab >2012-12-17 09:26:21,961 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [5ecdfb51] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:26:22,125 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [47111e6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:26:22,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47198520 >2012-12-17 09:26:22,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 759ea4d7 >2012-12-17 09:26:22,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:23,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, SpmStopVDSCommand, log id: 759ea4d7 >2012-12-17 09:26:23,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, ResetIrsVDSCommand, log id: 47198520 >2012-12-17 09:26:23,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8332b77 >2012-12-17 09:26:23,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:23,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8332b77 >2012-12-17 09:26:23,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:26:23,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4854 >2012-12-17 09:26:23,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4854, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 695e79db >2012-12-17 09:26:23,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling started: taskId = 6e015609-4f46-480e-a72e-41e80c606c2c >2012-12-17 09:26:24,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling ended: taskId = 6e015609-4f46-480e-a72e-41e80c606c2c task status = finished >2012-12-17 09:26:24,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] spmStart polling ended, spm status: SPM >2012-12-17 09:26:24,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6e015609-4f46-480e-a72e-41e80c606c2c), log id: 295ac09d >2012-12-17 09:26:24,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMClearTaskVDSCommand, log id: 295ac09d >2012-12-17 09:26:24,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56c174a5, log id: 695e79db >2012-12-17 09:26:24,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:26:24,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:24,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1968713b >2012-12-17 09:26:24,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1968713b >2012-12-17 09:26:24,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b7532ab >2012-12-17 09:26:24,430 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:24,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 17e1e585 >2012-12-17 09:26:24,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:24,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78474385 >2012-12-17 09:26:24,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78474385 >2012-12-17 09:26:24,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [3aebafd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17e1e585 >2012-12-17 09:26:24,450 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [3aebafd8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:33,331 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [67dedb1b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:26:33,570 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [3fe36f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:26:33,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f91f287 >2012-12-17 09:26:33,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e24560f >2012-12-17 09:26:33,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:34,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SpmStopVDSCommand, log id: 7e24560f >2012-12-17 09:26:34,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, ResetIrsVDSCommand, log id: 6f91f287 >2012-12-17 09:26:34,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7596f185 >2012-12-17 09:26:34,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:34,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7596f185 >2012-12-17 09:26:34,336 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:26:44,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:26:44,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4855 >2012-12-17 09:26:44,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4855, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64234e4d >2012-12-17 09:26:44,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling started: taskId = cb212e4c-b7b4-44f6-bec6-ee0d683bc06c >2012-12-17 09:26:45,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling ended: taskId = cb212e4c-b7b4-44f6-bec6-ee0d683bc06c task status = finished >2012-12-17 09:26:45,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] spmStart polling ended, spm status: SPM >2012-12-17 09:26:45,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb212e4c-b7b4-44f6-bec6-ee0d683bc06c), log id: 7066ac4f >2012-12-17 09:26:45,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, HSMClearTaskVDSCommand, log id: 7066ac4f >2012-12-17 09:26:45,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [48bd7fc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64bab02f, log id: 64234e4d >2012-12-17 09:26:45,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:26:46,461 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:26:46,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a2c195c >2012-12-17 09:26:46,741 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [48bd7fc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:26:47,035 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [5cb68cfd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:26:47,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b65fc07 >2012-12-17 09:26:47,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24b4d600 >2012-12-17 09:26:47,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:47,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, SpmStopVDSCommand, log id: 24b4d600 >2012-12-17 09:26:47,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, ResetIrsVDSCommand, log id: 1b65fc07 >2012-12-17 09:26:47,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c8b3dea >2012-12-17 09:26:47,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5cb68cfd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:47,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [5cb68cfd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c8b3dea >2012-12-17 09:26:47,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:26:47,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4856 >2012-12-17 09:26:47,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4856, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2168c598 >2012-12-17 09:26:47,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling started: taskId = 657e11ff-4690-43cf-bd82-43021a7e4245 >2012-12-17 09:26:48,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling ended: taskId = 657e11ff-4690-43cf-bd82-43021a7e4245 task status = finished >2012-12-17 09:26:48,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] spmStart polling ended, spm status: SPM >2012-12-17 09:26:48,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=657e11ff-4690-43cf-bd82-43021a7e4245), log id: 772bcdb8 >2012-12-17 09:26:48,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMClearTaskVDSCommand, log id: 772bcdb8 >2012-12-17 09:26:48,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ccf2827, log id: 2168c598 >2012-12-17 09:26:48,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:26:48,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:48,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74fca4a6 >2012-12-17 09:26:48,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74fca4a6 >2012-12-17 09:26:48,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a2c195c >2012-12-17 09:26:48,556 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:48,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 776f31ec >2012-12-17 09:26:48,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:48,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1510b0a6 >2012-12-17 09:26:48,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1510b0a6 >2012-12-17 09:26:48,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-56) [733324dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 776f31ec >2012-12-17 09:26:48,581 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-56) [733324dc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:26:54,894 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:26:57,427 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [7bb72f96] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:26:57,613 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [4f4f0222] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:26:57,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5132efe9 >2012-12-17 09:26:57,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d5555af >2012-12-17 09:26:57,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:58,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SpmStopVDSCommand, log id: 7d5555af >2012-12-17 09:26:58,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, ResetIrsVDSCommand, log id: 5132efe9 >2012-12-17 09:26:58,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 133428bc >2012-12-17 09:26:58,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:26:58,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 133428bc >2012-12-17 09:27:05,101 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:27:08,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:08,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4857 >2012-12-17 09:27:08,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4857, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 17f38d43 >2012-12-17 09:27:08,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling started: taskId = 4f431e6b-9fba-476e-a5de-b620c6d7bb9e >2012-12-17 09:27:09,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling ended: taskId = 4f431e6b-9fba-476e-a5de-b620c6d7bb9e task status = finished >2012-12-17 09:27:09,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] spmStart polling ended, spm status: SPM >2012-12-17 09:27:09,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4f431e6b-9fba-476e-a5de-b620c6d7bb9e), log id: 58d4409f >2012-12-17 09:27:09,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, HSMClearTaskVDSCommand, log id: 58d4409f >2012-12-17 09:27:09,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [2a365c8d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56deb448, log id: 17f38d43 >2012-12-17 09:27:09,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:09,380 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:27:09,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51e12ba5 >2012-12-17 09:27:09,402 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [2a365c8d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:27:09,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:27:09,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 368222fb >2012-12-17 09:27:09,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 39467503 >2012-12-17 09:27:09,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:09,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, SpmStopVDSCommand, log id: 39467503 >2012-12-17 09:27:10,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, ResetIrsVDSCommand, log id: 368222fb >2012-12-17 09:27:10,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10e95887 >2012-12-17 09:27:10,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:10,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10e95887 >2012-12-17 09:27:10,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:10,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4858 >2012-12-17 09:27:10,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4858, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39f92493 >2012-12-17 09:27:10,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 52e705fd-8f0c-4443-90a9-8dcd7bb5d720 >2012-12-17 09:27:11,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 52e705fd-8f0c-4443-90a9-8dcd7bb5d720 task status = finished >2012-12-17 09:27:11,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM >2012-12-17 09:27:11,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=52e705fd-8f0c-4443-90a9-8dcd7bb5d720), log id: 23a6ab00 >2012-12-17 09:27:11,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: 23a6ab00 >2012-12-17 09:27:11,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ff9b88a, log id: 39f92493 >2012-12-17 09:27:11,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:11,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:11,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6accbc68 >2012-12-17 09:27:11,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6accbc68 >2012-12-17 09:27:11,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51e12ba5 >2012-12-17 09:27:11,407 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:11,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2765869f >2012-12-17 09:27:11,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:11,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 504ee3c3 >2012-12-17 09:27:11,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 504ee3c3 >2012-12-17 09:27:11,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2765869f >2012-12-17 09:27:11,428 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:15,365 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:27:20,123 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4d69cef1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:27:20,264 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [651c3e82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:27:20,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 52d0fb70 >2012-12-17 09:27:20,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6023f5d9 >2012-12-17 09:27:20,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:20,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SpmStopVDSCommand, log id: 6023f5d9 >2012-12-17 09:27:21,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, ResetIrsVDSCommand, log id: 52d0fb70 >2012-12-17 09:27:21,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c6ed2a7 >2012-12-17 09:27:21,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:21,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c6ed2a7 >2012-12-17 09:27:26,978 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:27:31,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:31,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4859 >2012-12-17 09:27:31,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4859, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d6eac23 >2012-12-17 09:27:31,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling started: taskId = bc919962-55b5-41a3-8e3a-a8bfd7c4c601 >2012-12-17 09:27:32,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling ended: taskId = bc919962-55b5-41a3-8e3a-a8bfd7c4c601 task status = finished >2012-12-17 09:27:32,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] spmStart polling ended, spm status: SPM >2012-12-17 09:27:32,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc919962-55b5-41a3-8e3a-a8bfd7c4c601), log id: 60279fc5 >2012-12-17 09:27:32,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, HSMClearTaskVDSCommand, log id: 60279fc5 >2012-12-17 09:27:32,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [7e80f448] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@496a6fd4, log id: 2d6eac23 >2012-12-17 09:27:32,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:32,598 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:27:32,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42cc891b >2012-12-17 09:27:32,614 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7e80f448] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:27:32,742 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [65f4d22d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:27:32,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36a0cfa1 >2012-12-17 09:27:32,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 242d6a1c >2012-12-17 09:27:32,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:32,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, SpmStopVDSCommand, log id: 242d6a1c >2012-12-17 09:27:32,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, ResetIrsVDSCommand, log id: 36a0cfa1 >2012-12-17 09:27:32,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75461654 >2012-12-17 09:27:32,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:32,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75461654 >2012-12-17 09:27:33,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:33,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4860 >2012-12-17 09:27:33,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4860, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 192e9b9e >2012-12-17 09:27:33,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = 1f915299-d10a-4173-96ab-f35b3022cf1c >2012-12-17 09:27:34,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = 1f915299-d10a-4173-96ab-f35b3022cf1c task status = finished >2012-12-17 09:27:34,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM >2012-12-17 09:27:34,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1f915299-d10a-4173-96ab-f35b3022cf1c), log id: 7f345cf9 >2012-12-17 09:27:34,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 7f345cf9 >2012-12-17 09:27:34,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46c420ad, log id: 192e9b9e >2012-12-17 09:27:34,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:34,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:34,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: fc5d975 >2012-12-17 09:27:34,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: fc5d975 >2012-12-17 09:27:34,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42cc891b >2012-12-17 09:27:34,201 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:34,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46675a5e >2012-12-17 09:27:34,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:34,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79510662 >2012-12-17 09:27:34,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79510662 >2012-12-17 09:27:34,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46675a5e >2012-12-17 09:27:34,221 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-58) [7a1c9fc6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:37,209 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:27:43,074 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [94e4f4d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:27:43,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [5c496b37] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:27:43,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45ca6579 >2012-12-17 09:27:43,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ca65381 >2012-12-17 09:27:43,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:43,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStopVDSCommand, log id: ca65381 >2012-12-17 09:27:43,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, ResetIrsVDSCommand, log id: 45ca6579 >2012-12-17 09:27:43,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2da3cf6f >2012-12-17 09:27:43,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:43,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2da3cf6f >2012-12-17 09:27:47,476 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:27:53,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:53,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4861 >2012-12-17 09:27:53,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4861, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 9ec387c >2012-12-17 09:27:53,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling started: taskId = 19391ac2-88c6-4eaf-802a-deacc72c37db >2012-12-17 09:27:54,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling ended: taskId = 19391ac2-88c6-4eaf-802a-deacc72c37db task status = finished >2012-12-17 09:27:54,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] spmStart polling ended, spm status: SPM >2012-12-17 09:27:54,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=19391ac2-88c6-4eaf-802a-deacc72c37db), log id: 2a6460cb >2012-12-17 09:27:54,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, HSMClearTaskVDSCommand, log id: 2a6460cb >2012-12-17 09:27:54,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [2a7b65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@240ec52b, log id: 9ec387c >2012-12-17 09:27:54,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:54,889 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:27:54,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27b39dc5 >2012-12-17 09:27:54,917 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [2a7b65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:27:55,106 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [d7157bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:27:55,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 66f5d546 >2012-12-17 09:27:55,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5db84d9 >2012-12-17 09:27:55,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:55,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, SpmStopVDSCommand, log id: 5db84d9 >2012-12-17 09:27:55,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, ResetIrsVDSCommand, log id: 66f5d546 >2012-12-17 09:27:55,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4889b2bf >2012-12-17 09:27:55,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d7157bb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:55,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d7157bb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4889b2bf >2012-12-17 09:27:55,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:27:55,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4862 >2012-12-17 09:27:55,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4862, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68097485 >2012-12-17 09:27:55,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling started: taskId = 35132a69-ac3b-480a-bfca-2f1b1ff36f4c >2012-12-17 09:27:56,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling ended: taskId = 35132a69-ac3b-480a-bfca-2f1b1ff36f4c task status = finished >2012-12-17 09:27:56,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] spmStart polling ended, spm status: SPM >2012-12-17 09:27:56,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=35132a69-ac3b-480a-bfca-2f1b1ff36f4c), log id: 440058f5 >2012-12-17 09:27:56,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMClearTaskVDSCommand, log id: 440058f5 >2012-12-17 09:27:56,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72d05fb3, log id: 68097485 >2012-12-17 09:27:56,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:27:56,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:56,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 53db6f53 >2012-12-17 09:27:56,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 53db6f53 >2012-12-17 09:27:56,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27b39dc5 >2012-12-17 09:27:56,712 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:56,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b69dd27 >2012-12-17 09:27:56,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:27:56,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b76e567 >2012-12-17 09:27:56,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b76e567 >2012-12-17 09:27:56,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [13d7977] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b69dd27 >2012-12-17 09:27:56,736 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [13d7977] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:27:57,727 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:28:05,437 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [327be49e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:28:06,455 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [6e561ed4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:28:06,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23578fa >2012-12-17 09:28:06,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7123380e >2012-12-17 09:28:06,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:06,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, SpmStopVDSCommand, log id: 7123380e >2012-12-17 09:28:06,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, ResetIrsVDSCommand, log id: 23578fa >2012-12-17 09:28:06,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7aca8471 >2012-12-17 09:28:06,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:06,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7aca8471 >2012-12-17 09:28:07,949 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:28:16,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:28:16,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4863 >2012-12-17 09:28:16,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4863, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bf5e53e >2012-12-17 09:28:16,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling started: taskId = d2640904-9461-4f4b-a96e-0f694acbeb31 >2012-12-17 09:28:17,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended: taskId = d2640904-9461-4f4b-a96e-0f694acbeb31 task status = finished >2012-12-17 09:28:17,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] spmStart polling ended, spm status: SPM >2012-12-17 09:28:17,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d2640904-9461-4f4b-a96e-0f694acbeb31), log id: 6f7910df >2012-12-17 09:28:17,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, HSMClearTaskVDSCommand, log id: 6f7910df >2012-12-17 09:28:17,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2155afb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7542d9a5, log id: 1bf5e53e >2012-12-17 09:28:17,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:28:17,941 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:28:17,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f690623 >2012-12-17 09:28:17,978 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2155afb6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:28:18,274 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [2c554bbe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:28:18,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47cdaf9b >2012-12-17 09:28:18,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 619605e0 >2012-12-17 09:28:18,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:18,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStopVDSCommand, log id: 619605e0 >2012-12-17 09:28:18,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, ResetIrsVDSCommand, log id: 47cdaf9b >2012-12-17 09:28:18,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10738131 >2012-12-17 09:28:18,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:18,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10738131 >2012-12-17 09:28:18,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:28:18,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4864 >2012-12-17 09:28:18,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4864, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4286753c >2012-12-17 09:28:18,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 36317878-b0bf-4593-a56e-c80cfbaa58c5 >2012-12-17 09:28:19,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 36317878-b0bf-4593-a56e-c80cfbaa58c5 task status = finished >2012-12-17 09:28:19,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM >2012-12-17 09:28:19,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=36317878-b0bf-4593-a56e-c80cfbaa58c5), log id: 47546308 >2012-12-17 09:28:19,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 47546308 >2012-12-17 09:28:19,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7531c6cd, log id: 4286753c >2012-12-17 09:28:19,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:28:19,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:19,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f05c950 >2012-12-17 09:28:19,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f05c950 >2012-12-17 09:28:19,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f690623 >2012-12-17 09:28:19,749 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:28:19,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52bbe1bd >2012-12-17 09:28:19,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:19,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 298ed541 >2012-12-17 09:28:19,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 298ed541 >2012-12-17 09:28:19,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52bbe1bd >2012-12-17 09:28:19,770 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:28:28,437 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:28:28,627 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [efbb817] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:28:28,796 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:28:28,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: df3cd3b >2012-12-17 09:28:28,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49aca843 >2012-12-17 09:28:28,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:29,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, SpmStopVDSCommand, log id: 49aca843 >2012-12-17 09:28:29,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, ResetIrsVDSCommand, log id: df3cd3b >2012-12-17 09:28:29,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7489490c >2012-12-17 09:28:29,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:29,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [5c6bc06e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7489490c >2012-12-17 09:28:38,682 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:28:39,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:28:39,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4865 >2012-12-17 09:28:39,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4865, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31ac78f >2012-12-17 09:28:39,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling started: taskId = cb8c0197-0618-4096-b0fd-5f2555f6c005 >2012-12-17 09:28:40,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling ended: taskId = cb8c0197-0618-4096-b0fd-5f2555f6c005 task status = finished >2012-12-17 09:28:40,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] spmStart polling ended, spm status: SPM >2012-12-17 09:28:40,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cb8c0197-0618-4096-b0fd-5f2555f6c005), log id: 213b0f3d >2012-12-17 09:28:40,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, HSMClearTaskVDSCommand, log id: 213b0f3d >2012-12-17 09:28:40,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5d07077e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c657ed6, log id: 31ac78f >2012-12-17 09:28:41,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:28:41,251 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:28:41,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fb11eb6 >2012-12-17 09:28:41,277 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5d07077e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:28:41,402 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [be98275] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:28:41,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4cabb600 >2012-12-17 09:28:41,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 73ebdae8 >2012-12-17 09:28:41,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:42,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, SpmStopVDSCommand, log id: 73ebdae8 >2012-12-17 09:28:42,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, ResetIrsVDSCommand, log id: 4cabb600 >2012-12-17 09:28:42,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 705c5f97 >2012-12-17 09:28:42,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:42,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 705c5f97 >2012-12-17 09:28:42,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:28:42,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4866 >2012-12-17 09:28:42,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4866, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b49b323 >2012-12-17 09:28:42,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling started: taskId = faa598b3-15c3-48ea-9546-d744dc26eeb7 >2012-12-17 09:28:43,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling ended: taskId = faa598b3-15c3-48ea-9546-d744dc26eeb7 task status = finished >2012-12-17 09:28:43,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] spmStart polling ended, spm status: SPM >2012-12-17 09:28:43,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=faa598b3-15c3-48ea-9546-d744dc26eeb7), log id: def2b43 >2012-12-17 09:28:43,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMClearTaskVDSCommand, log id: def2b43 >2012-12-17 09:28:43,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@510142da, log id: 4b49b323 >2012-12-17 09:28:43,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:28:43,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:43,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 23554bb6 >2012-12-17 09:28:43,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23554bb6 >2012-12-17 09:28:43,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fb11eb6 >2012-12-17 09:28:43,502 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:28:43,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34ddabd9 >2012-12-17 09:28:43,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:43,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 602e5ddf >2012-12-17 09:28:43,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 602e5ddf >2012-12-17 09:28:43,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [5389729] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34ddabd9 >2012-12-17 09:28:43,522 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [5389729] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:28:48,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:28:52,331 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [5ab536e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:28:52,431 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [136706cf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:28:52,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6af4f54b >2012-12-17 09:28:52,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 336ed08c >2012-12-17 09:28:52,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:52,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, SpmStopVDSCommand, log id: 336ed08c >2012-12-17 09:28:52,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, ResetIrsVDSCommand, log id: 6af4f54b >2012-12-17 09:28:52,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [136706cf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 482dc65c >2012-12-17 09:28:52,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:28:52,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [136706cf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 482dc65c >2012-12-17 09:28:59,109 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:29:02,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:02,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4867 >2012-12-17 09:29:02,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4867, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59da520c >2012-12-17 09:29:02,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling started: taskId = f9ee22a5-4d03-41cf-be07-f912a5ee33c4 >2012-12-17 09:29:03,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended: taskId = f9ee22a5-4d03-41cf-be07-f912a5ee33c4 task status = finished >2012-12-17 09:29:03,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] spmStart polling ended, spm status: SPM >2012-12-17 09:29:03,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9ee22a5-4d03-41cf-be07-f912a5ee33c4), log id: a741db6 >2012-12-17 09:29:03,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, HSMClearTaskVDSCommand, log id: a741db6 >2012-12-17 09:29:03,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [1925ecb1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d4a9d16, log id: 59da520c >2012-12-17 09:29:03,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:03,931 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:29:03,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9a2a5e7 >2012-12-17 09:29:03,948 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1925ecb1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:29:04,076 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [56a0d824] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:29:04,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 269beee5 >2012-12-17 09:29:04,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2113d277 >2012-12-17 09:29:04,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:04,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, SpmStopVDSCommand, log id: 2113d277 >2012-12-17 09:29:04,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, ResetIrsVDSCommand, log id: 269beee5 >2012-12-17 09:29:04,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 394264f4 >2012-12-17 09:29:04,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:04,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 394264f4 >2012-12-17 09:29:04,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:04,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4868 >2012-12-17 09:29:04,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4868, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d25b0fb >2012-12-17 09:29:04,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling started: taskId = 1f705429-cb32-476b-a594-048a9601f738 >2012-12-17 09:29:05,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended: taskId = 1f705429-cb32-476b-a594-048a9601f738 task status = finished >2012-12-17 09:29:05,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended, spm status: SPM >2012-12-17 09:29:05,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1f705429-cb32-476b-a594-048a9601f738), log id: 33776c1a >2012-12-17 09:29:05,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMClearTaskVDSCommand, log id: 33776c1a >2012-12-17 09:29:05,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@655a395b, log id: 6d25b0fb >2012-12-17 09:29:06,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:06,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:06,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6896349a >2012-12-17 09:29:06,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6896349a >2012-12-17 09:29:06,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9a2a5e7 >2012-12-17 09:29:06,308 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:06,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5d57a01a >2012-12-17 09:29:06,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:06,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 37d762fc >2012-12-17 09:29:06,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 37d762fc >2012-12-17 09:29:06,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5d57a01a >2012-12-17 09:29:06,328 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [160c510e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:09,285 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:29:14,468 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [1cf2f3eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:29:14,651 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [7a6ed577] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:29:14,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 72c5c564 >2012-12-17 09:29:14,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e12c6bf >2012-12-17 09:29:14,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:14,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStopVDSCommand, log id: 3e12c6bf >2012-12-17 09:29:14,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, ResetIrsVDSCommand, log id: 72c5c564 >2012-12-17 09:29:14,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74b98e56 >2012-12-17 09:29:14,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:14,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74b98e56 >2012-12-17 09:29:19,549 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:29:25,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:25,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4869 >2012-12-17 09:29:25,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4869, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 359e225d >2012-12-17 09:29:25,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling started: taskId = 876570e6-055f-4b7e-8e1d-b6ebfcc2d988 >2012-12-17 09:29:26,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended: taskId = 876570e6-055f-4b7e-8e1d-b6ebfcc2d988 task status = finished >2012-12-17 09:29:26,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] spmStart polling ended, spm status: SPM >2012-12-17 09:29:26,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=876570e6-055f-4b7e-8e1d-b6ebfcc2d988), log id: 4248830a >2012-12-17 09:29:26,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, HSMClearTaskVDSCommand, log id: 4248830a >2012-12-17 09:29:26,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [71b7bdef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26932785, log id: 359e225d >2012-12-17 09:29:26,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:26,711 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:29:26,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 311ed19c >2012-12-17 09:29:26,761 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [71b7bdef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:29:26,857 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [280f54d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:29:26,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17716e9c >2012-12-17 09:29:26,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15995375 >2012-12-17 09:29:26,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:27,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, SpmStopVDSCommand, log id: 15995375 >2012-12-17 09:29:27,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, ResetIrsVDSCommand, log id: 17716e9c >2012-12-17 09:29:27,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fff36b2 >2012-12-17 09:29:27,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:27,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fff36b2 >2012-12-17 09:29:27,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:27,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4870 >2012-12-17 09:29:27,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4870, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f70bbe2 >2012-12-17 09:29:27,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling started: taskId = ce30fad2-2c85-42bf-a79a-6fe585a2b296 >2012-12-17 09:29:28,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended: taskId = ce30fad2-2c85-42bf-a79a-6fe585a2b296 task status = finished >2012-12-17 09:29:28,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended, spm status: SPM >2012-12-17 09:29:28,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ce30fad2-2c85-42bf-a79a-6fe585a2b296), log id: 72ca5e04 >2012-12-17 09:29:28,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMClearTaskVDSCommand, log id: 72ca5e04 >2012-12-17 09:29:28,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43e2937c, log id: 3f70bbe2 >2012-12-17 09:29:28,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:28,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:28,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 383ec615 >2012-12-17 09:29:28,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 383ec615 >2012-12-17 09:29:28,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 311ed19c >2012-12-17 09:29:28,744 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:28,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 213ea98c >2012-12-17 09:29:28,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:28,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25e52a39 >2012-12-17 09:29:28,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25e52a39 >2012-12-17 09:29:28,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 213ea98c >2012-12-17 09:29:28,764 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:29,735 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:29:37,587 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3aebafd8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:29:37,734 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [42560f9e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:29:37,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65232a6a >2012-12-17 09:29:37,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2238829b >2012-12-17 09:29:37,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:38,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStopVDSCommand, log id: 2238829b >2012-12-17 09:29:38,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, ResetIrsVDSCommand, log id: 65232a6a >2012-12-17 09:29:38,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d17e67b >2012-12-17 09:29:38,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:38,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d17e67b >2012-12-17 09:29:39,973 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:29:48,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:48,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4871 >2012-12-17 09:29:48,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4871, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dd8977e >2012-12-17 09:29:48,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling started: taskId = f1d34bf6-dadb-4387-ae8f-f72a91918a57 >2012-12-17 09:29:49,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling ended: taskId = f1d34bf6-dadb-4387-ae8f-f72a91918a57 task status = finished >2012-12-17 09:29:49,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] spmStart polling ended, spm status: SPM >2012-12-17 09:29:49,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1d34bf6-dadb-4387-ae8f-f72a91918a57), log id: 29f52a31 >2012-12-17 09:29:49,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, HSMClearTaskVDSCommand, log id: 29f52a31 >2012-12-17 09:29:49,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [1cfebcc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ea50c7c, log id: 2dd8977e >2012-12-17 09:29:49,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:49,343 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:29:49,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 37d1bbbf >2012-12-17 09:29:49,357 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [1cfebcc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:29:49,500 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [6c33f0df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:29:49,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 35eed4b4 >2012-12-17 09:29:49,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bbe9688 >2012-12-17 09:29:49,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:49,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, SpmStopVDSCommand, log id: 3bbe9688 >2012-12-17 09:29:49,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, ResetIrsVDSCommand, log id: 35eed4b4 >2012-12-17 09:29:49,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 16768ea2 >2012-12-17 09:29:49,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:49,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 16768ea2 >2012-12-17 09:29:49,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:29:49,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4872 >2012-12-17 09:29:49,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4872, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e2efb3 >2012-12-17 09:29:49,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = a3230d65-581c-4547-a8b7-e43ff39d0532 >2012-12-17 09:29:50,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = a3230d65-581c-4547-a8b7-e43ff39d0532 task status = finished >2012-12-17 09:29:50,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM >2012-12-17 09:29:50,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3230d65-581c-4547-a8b7-e43ff39d0532), log id: 180316fc >2012-12-17 09:29:50,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 180316fc >2012-12-17 09:29:50,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72da06b4, log id: 15e2efb3 >2012-12-17 09:29:51,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:29:51,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:51,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 459aff85 >2012-12-17 09:29:51,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 459aff85 >2012-12-17 09:29:51,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 37d1bbbf >2012-12-17 09:29:51,578 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:51,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42fd9876 >2012-12-17 09:29:51,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:29:51,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5362a7ed >2012-12-17 09:29:51,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5362a7ed >2012-12-17 09:29:51,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42fd9876 >2012-12-17 09:29:51,598 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:29:59,861 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [733324dc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Checking autorecoverable hosts >2012-12-17 09:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Autorecovering 0 hosts >2012-12-17 09:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Checking autorecoverable hosts done >2012-12-17 09:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [3a1c2706] Autorecovering storage domains is disabled, skipping >2012-12-17 09:30:00,030 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [6de308ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:00,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d973d6f >2012-12-17 09:30:00,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 189253e9 >2012-12-17 09:30:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, SpmStopVDSCommand, log id: 189253e9 >2012-12-17 09:30:00,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, ResetIrsVDSCommand, log id: 7d973d6f >2012-12-17 09:30:00,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f20907d >2012-12-17 09:30:00,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:00,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f20907d >2012-12-17 09:30:00,611 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:30:11,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:11,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4873 >2012-12-17 09:30:11,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4873, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2deba123 >2012-12-17 09:30:11,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling started: taskId = ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e >2012-12-17 09:30:12,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling ended: taskId = ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e task status = finished >2012-12-17 09:30:12,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] spmStart polling ended, spm status: SPM >2012-12-17 09:30:12,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ef3b1a6c-e028-4bf9-b681-e4f8fe1a9b7e), log id: 7226118a >2012-12-17 09:30:12,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, HSMClearTaskVDSCommand, log id: 7226118a >2012-12-17 09:30:12,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [223c8c2d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2149b592, log id: 2deba123 >2012-12-17 09:30:12,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:30:12,627 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:30:12,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ebf4809 >2012-12-17 09:30:12,665 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [223c8c2d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:12,755 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [1813139a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:12,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 71ba17fd >2012-12-17 09:30:12,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26a4fc03 >2012-12-17 09:30:12,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:13,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, SpmStopVDSCommand, log id: 26a4fc03 >2012-12-17 09:30:13,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, ResetIrsVDSCommand, log id: 71ba17fd >2012-12-17 09:30:13,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [1813139a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d9d0960 >2012-12-17 09:30:13,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [1813139a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:13,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [1813139a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d9d0960 >2012-12-17 09:30:13,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:13,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4874 >2012-12-17 09:30:13,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4874, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a927837 >2012-12-17 09:30:13,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling started: taskId = 4d2aa85d-ba88-4a87-b26d-4c01694dd3c7 >2012-12-17 09:30:14,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling ended: taskId = 4d2aa85d-ba88-4a87-b26d-4c01694dd3c7 task status = finished >2012-12-17 09:30:14,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] spmStart polling ended, spm status: SPM >2012-12-17 09:30:14,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d2aa85d-ba88-4a87-b26d-4c01694dd3c7), log id: 48053e9c >2012-12-17 09:30:14,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMClearTaskVDSCommand, log id: 48053e9c >2012-12-17 09:30:14,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@505198b4, log id: a927837 >2012-12-17 09:30:14,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:30:14,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:14,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 10332a07 >2012-12-17 09:30:14,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10332a07 >2012-12-17 09:30:14,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ebf4809 >2012-12-17 09:30:14,422 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:30:14,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6baba8d7 >2012-12-17 09:30:14,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:14,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5c804efb >2012-12-17 09:30:14,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5c804efb >2012-12-17 09:30:14,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [20c0de80] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6baba8d7 >2012-12-17 09:30:14,442 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [20c0de80] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:30:22,153 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:30:23,313 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [10684b21] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:23,455 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [7db92791] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:23,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7614816d >2012-12-17 09:30:23,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44a4be41 >2012-12-17 09:30:23,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:23,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SpmStopVDSCommand, log id: 44a4be41 >2012-12-17 09:30:23,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, ResetIrsVDSCommand, log id: 7614816d >2012-12-17 09:30:23,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31bb0dbb >2012-12-17 09:30:23,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:23,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31bb0dbb >2012-12-17 09:30:32,363 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:30:33,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:33,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4875 >2012-12-17 09:30:33,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4875, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 101d8222 >2012-12-17 09:30:33,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling started: taskId = dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534 >2012-12-17 09:30:34,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling ended: taskId = dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534 task status = finished >2012-12-17 09:30:34,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] spmStart polling ended, spm status: SPM >2012-12-17 09:30:34,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dacd1950-5ed6-4ac7-a7ef-8d6ca9b72534), log id: ea3dbe0 >2012-12-17 09:30:34,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, HSMClearTaskVDSCommand, log id: ea3dbe0 >2012-12-17 09:30:34,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [3c6804c8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27b856a2, log id: 101d8222 >2012-12-17 09:30:34,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:30:35,015 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:30:35,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5588918a >2012-12-17 09:30:35,030 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3c6804c8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:35,154 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [603633ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:35,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1008b992 >2012-12-17 09:30:35,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49c61278 >2012-12-17 09:30:35,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:35,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, SpmStopVDSCommand, log id: 49c61278 >2012-12-17 09:30:35,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, ResetIrsVDSCommand, log id: 1008b992 >2012-12-17 09:30:35,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [603633ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d29ac82 >2012-12-17 09:30:35,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [603633ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:35,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [603633ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d29ac82 >2012-12-17 09:30:36,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:36,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4876 >2012-12-17 09:30:36,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4876, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e42a310 >2012-12-17 09:30:36,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling started: taskId = 70362b6b-0b4d-438f-a1d1-2ddcbe489210 >2012-12-17 09:30:37,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling ended: taskId = 70362b6b-0b4d-438f-a1d1-2ddcbe489210 task status = finished >2012-12-17 09:30:37,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] spmStart polling ended, spm status: SPM >2012-12-17 09:30:37,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70362b6b-0b4d-438f-a1d1-2ddcbe489210), log id: 57daa2ec >2012-12-17 09:30:37,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMClearTaskVDSCommand, log id: 57daa2ec >2012-12-17 09:30:37,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52fe6103, log id: 5e42a310 >2012-12-17 09:30:37,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:30:37,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:37,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4de540eb >2012-12-17 09:30:37,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4de540eb >2012-12-17 09:30:37,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5588918a >2012-12-17 09:30:37,675 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:30:37,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6092c41 >2012-12-17 09:30:37,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:37,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2db24135 >2012-12-17 09:30:37,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2db24135 >2012-12-17 09:30:37,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-41) [26c9f399] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6092c41 >2012-12-17 09:30:37,696 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-41) [26c9f399] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:30:42,650 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:30:46,445 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5cb68cfd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:46,811 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [462eb8a5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:46,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 817854 >2012-12-17 09:30:46,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 27b1e2e1 >2012-12-17 09:30:46,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:47,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SpmStopVDSCommand, log id: 27b1e2e1 >2012-12-17 09:30:47,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, ResetIrsVDSCommand, log id: 817854 >2012-12-17 09:30:47,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 575602c0 >2012-12-17 09:30:47,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:47,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 575602c0 >2012-12-17 09:30:52,886 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:30:57,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:57,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4877 >2012-12-17 09:30:57,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4877, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50984a60 >2012-12-17 09:30:57,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling started: taskId = 0cf67029-1890-40be-aa40-c97caeae5775 >2012-12-17 09:30:58,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling ended: taskId = 0cf67029-1890-40be-aa40-c97caeae5775 task status = finished >2012-12-17 09:30:58,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] spmStart polling ended, spm status: SPM >2012-12-17 09:30:58,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0cf67029-1890-40be-aa40-c97caeae5775), log id: 2b980dea >2012-12-17 09:30:58,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, HSMClearTaskVDSCommand, log id: 2b980dea >2012-12-17 09:30:58,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [550a1bf7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@bb99402, log id: 50984a60 >2012-12-17 09:30:58,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:30:58,358 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:30:58,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ae4cf34 >2012-12-17 09:30:58,407 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [550a1bf7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:30:58,600 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [e8edb34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:30:58,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76bdbdd2 >2012-12-17 09:30:58,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5cccfd6f >2012-12-17 09:30:58,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:58,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, SpmStopVDSCommand, log id: 5cccfd6f >2012-12-17 09:30:58,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, ResetIrsVDSCommand, log id: 76bdbdd2 >2012-12-17 09:30:58,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49db2046 >2012-12-17 09:30:58,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:30:58,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [e8edb34] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49db2046 >2012-12-17 09:30:58,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:30:58,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4878 >2012-12-17 09:30:58,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4878, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 140841e7 >2012-12-17 09:30:58,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling started: taskId = d9592c81-084f-4fd1-9b33-6f9a2c182860 >2012-12-17 09:30:59,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling ended: taskId = d9592c81-084f-4fd1-9b33-6f9a2c182860 task status = finished >2012-12-17 09:31:00,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] spmStart polling ended, spm status: SPM >2012-12-17 09:31:00,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d9592c81-084f-4fd1-9b33-6f9a2c182860), log id: 6997a10a >2012-12-17 09:31:00,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMClearTaskVDSCommand, log id: 6997a10a >2012-12-17 09:31:00,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61cdcdae, log id: 140841e7 >2012-12-17 09:31:00,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:31:00,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 68d89b75 >2012-12-17 09:31:00,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 68d89b75 >2012-12-17 09:31:00,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ae4cf34 >2012-12-17 09:31:00,095 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:00,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 64c65419 >2012-12-17 09:31:00,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:00,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6935c578 >2012-12-17 09:31:00,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6935c578 >2012-12-17 09:31:00,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [6e84ec96] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 64c65419 >2012-12-17 09:31:00,114 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [6e84ec96] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:03,083 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:31:08,964 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [a029edc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:31:09,091 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [6157bb5e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:31:09,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a62ce1e >2012-12-17 09:31:09,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74b5e2fe >2012-12-17 09:31:09,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:09,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SpmStopVDSCommand, log id: 74b5e2fe >2012-12-17 09:31:09,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, ResetIrsVDSCommand, log id: 6a62ce1e >2012-12-17 09:31:09,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ee643e2 >2012-12-17 09:31:09,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:09,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ee643e2 >2012-12-17 09:31:13,262 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:31:20,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:31:20,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4879 >2012-12-17 09:31:20,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4879, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ea7a045 >2012-12-17 09:31:20,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling started: taskId = 18c9d7a0-76bc-412e-b3e0-e227ba7fae9d >2012-12-17 09:31:21,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended: taskId = 18c9d7a0-76bc-412e-b3e0-e227ba7fae9d task status = finished >2012-12-17 09:31:21,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] spmStart polling ended, spm status: SPM >2012-12-17 09:31:21,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=18c9d7a0-76bc-412e-b3e0-e227ba7fae9d), log id: 6ac424e1 >2012-12-17 09:31:21,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, HSMClearTaskVDSCommand, log id: 6ac424e1 >2012-12-17 09:31:21,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [61136831] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73516689, log id: 4ea7a045 >2012-12-17 09:31:21,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:31:21,362 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:31:21,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2da985ca >2012-12-17 09:31:21,543 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [61136831] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:31:21,694 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [48ba699e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:31:21,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14c2c8b6 >2012-12-17 09:31:21,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 37e782b5 >2012-12-17 09:31:21,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:21,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, SpmStopVDSCommand, log id: 37e782b5 >2012-12-17 09:31:21,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, ResetIrsVDSCommand, log id: 14c2c8b6 >2012-12-17 09:31:21,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ecdd5bf >2012-12-17 09:31:21,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [48ba699e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:21,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [48ba699e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ecdd5bf >2012-12-17 09:31:21,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:31:22,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4880 >2012-12-17 09:31:22,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4880, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7125c62a >2012-12-17 09:31:22,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling started: taskId = 8458759d-4f5f-4d78-8a35-ed0b81652baf >2012-12-17 09:31:23,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended: taskId = 8458759d-4f5f-4d78-8a35-ed0b81652baf task status = finished >2012-12-17 09:31:23,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended, spm status: SPM >2012-12-17 09:31:23,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8458759d-4f5f-4d78-8a35-ed0b81652baf), log id: 7fa93d7b >2012-12-17 09:31:23,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMClearTaskVDSCommand, log id: 7fa93d7b >2012-12-17 09:31:23,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@400397b7, log id: 7125c62a >2012-12-17 09:31:23,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:31:23,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:23,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 947346a >2012-12-17 09:31:23,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 947346a >2012-12-17 09:31:23,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2da985ca >2012-12-17 09:31:23,141 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:23,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fbcc9a8 >2012-12-17 09:31:23,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:23,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a9c2c80 >2012-12-17 09:31:23,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a9c2c80 >2012-12-17 09:31:23,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fbcc9a8 >2012-12-17 09:31:23,161 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [e893ae] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:23,474 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:31:32,038 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [45dceb6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:31:32,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [134ba83e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:31:32,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 445553cc >2012-12-17 09:31:32,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 61c38280 >2012-12-17 09:31:32,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:32,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SpmStopVDSCommand, log id: 61c38280 >2012-12-17 09:31:32,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, ResetIrsVDSCommand, log id: 445553cc >2012-12-17 09:31:32,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fe58d95 >2012-12-17 09:31:32,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:32,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fe58d95 >2012-12-17 09:31:33,672 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:31:42,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:31:42,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4881 >2012-12-17 09:31:42,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4881, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d0484f9 >2012-12-17 09:31:42,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling started: taskId = 06cbe60a-038d-4526-8e13-73490e871d33 >2012-12-17 09:31:43,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling ended: taskId = 06cbe60a-038d-4526-8e13-73490e871d33 task status = finished >2012-12-17 09:31:43,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] spmStart polling ended, spm status: SPM >2012-12-17 09:31:43,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06cbe60a-038d-4526-8e13-73490e871d33), log id: 6f98481c >2012-12-17 09:31:43,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, HSMClearTaskVDSCommand, log id: 6f98481c >2012-12-17 09:31:43,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) [c8c45fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76a65f80, log id: 2d0484f9 >2012-12-17 09:31:43,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:31:43,779 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:31:43,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4bbbb5e5 >2012-12-17 09:31:43,798 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [c8c45fe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:31:43,986 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [3da48235] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:31:44,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24a263f1 >2012-12-17 09:31:44,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23a4ca5f >2012-12-17 09:31:44,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:44,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, SpmStopVDSCommand, log id: 23a4ca5f >2012-12-17 09:31:44,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, ResetIrsVDSCommand, log id: 24a263f1 >2012-12-17 09:31:44,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [3da48235] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40994e0a >2012-12-17 09:31:44,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [3da48235] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:44,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [3da48235] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40994e0a >2012-12-17 09:31:44,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:31:44,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4882 >2012-12-17 09:31:44,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4882, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a213bc7 >2012-12-17 09:31:44,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = eb9954f3-ab63-4574-9918-4507fd7bec02 >2012-12-17 09:31:45,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = eb9954f3-ab63-4574-9918-4507fd7bec02 task status = finished >2012-12-17 09:31:45,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM >2012-12-17 09:31:45,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eb9954f3-ab63-4574-9918-4507fd7bec02), log id: 233f43ce >2012-12-17 09:31:45,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 233f43ce >2012-12-17 09:31:45,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e7c9b76, log id: 4a213bc7 >2012-12-17 09:31:45,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:31:46,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:46,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6445f0 >2012-12-17 09:31:46,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6445f0 >2012-12-17 09:31:46,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4bbbb5e5 >2012-12-17 09:31:46,100 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:46,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7d2bb5b7 >2012-12-17 09:31:46,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:46,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 649610b6 >2012-12-17 09:31:46,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 649610b6 >2012-12-17 09:31:46,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7d2bb5b7 >2012-12-17 09:31:46,120 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) [1240becc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:31:54,156 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:31:54,782 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5389729] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:31:54,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [23d69744] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:31:55,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4162f15e >2012-12-17 09:31:55,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 555bc2a2 >2012-12-17 09:31:55,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:55,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, SpmStopVDSCommand, log id: 555bc2a2 >2012-12-17 09:31:56,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, ResetIrsVDSCommand, log id: 4162f15e >2012-12-17 09:31:56,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6be3c5b3 >2012-12-17 09:31:56,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:31:56,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6be3c5b3 >2012-12-17 09:32:04,333 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:32:06,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:06,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4883 >2012-12-17 09:32:06,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4883, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4576deca >2012-12-17 09:32:06,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling started: taskId = d7bd7d11-2850-44e4-b78f-0027d0df2adf >2012-12-17 09:32:07,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended: taskId = d7bd7d11-2850-44e4-b78f-0027d0df2adf task status = finished >2012-12-17 09:32:07,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] spmStart polling ended, spm status: SPM >2012-12-17 09:32:07,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7bd7d11-2850-44e4-b78f-0027d0df2adf), log id: 6b51eaef >2012-12-17 09:32:07,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, HSMClearTaskVDSCommand, log id: 6b51eaef >2012-12-17 09:32:07,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [779086b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f1727f4, log id: 4576deca >2012-12-17 09:32:07,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:07,960 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:32:07,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b7c75dd >2012-12-17 09:32:08,007 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [779086b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:32:08,196 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:32:08,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12dde100 >2012-12-17 09:32:08,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 791353dd >2012-12-17 09:32:08,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:08,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, SpmStopVDSCommand, log id: 791353dd >2012-12-17 09:32:08,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, ResetIrsVDSCommand, log id: 12dde100 >2012-12-17 09:32:08,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68c48d73 >2012-12-17 09:32:08,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:08,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68c48d73 >2012-12-17 09:32:08,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:09,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4884 >2012-12-17 09:32:09,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4884, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a5ec553 >2012-12-17 09:32:09,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling started: taskId = cfad4488-5339-43bc-bfef-efa7bcbe59ca >2012-12-17 09:32:10,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling ended: taskId = cfad4488-5339-43bc-bfef-efa7bcbe59ca task status = finished >2012-12-17 09:32:10,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] spmStart polling ended, spm status: SPM >2012-12-17 09:32:10,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cfad4488-5339-43bc-bfef-efa7bcbe59ca), log id: 373ed2e1 >2012-12-17 09:32:10,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMClearTaskVDSCommand, log id: 373ed2e1 >2012-12-17 09:32:10,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d86ce1c, log id: 3a5ec553 >2012-12-17 09:32:10,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:10,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:10,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 441b6e47 >2012-12-17 09:32:10,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 441b6e47 >2012-12-17 09:32:10,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b7c75dd >2012-12-17 09:32:10,168 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:32:10,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 64e51a3f >2012-12-17 09:32:10,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:10,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2e7ac27 >2012-12-17 09:32:10,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2e7ac27 >2012-12-17 09:32:10,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [3fe36f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 64e51a3f >2012-12-17 09:32:10,189 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [3fe36f5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:32:14,546 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:32:19,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [3b88f2b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:32:19,140 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [302432ce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:32:19,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a9b892e >2012-12-17 09:32:19,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: bbe5cc2 >2012-12-17 09:32:19,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:19,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStopVDSCommand, log id: bbe5cc2 >2012-12-17 09:32:19,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, ResetIrsVDSCommand, log id: 4a9b892e >2012-12-17 09:32:19,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d3e7bef >2012-12-17 09:32:19,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:19,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d3e7bef >2012-12-17 09:32:24,761 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:32:29,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:29,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4885 >2012-12-17 09:32:29,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4885, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58ee0f0c >2012-12-17 09:32:29,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling started: taskId = ab5ab054-c822-49ec-9ae0-62ec47da7210 >2012-12-17 09:32:30,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended: taskId = ab5ab054-c822-49ec-9ae0-62ec47da7210 task status = finished >2012-12-17 09:32:30,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] spmStart polling ended, spm status: SPM >2012-12-17 09:32:30,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab5ab054-c822-49ec-9ae0-62ec47da7210), log id: 3c693cd2 >2012-12-17 09:32:30,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, HSMClearTaskVDSCommand, log id: 3c693cd2 >2012-12-17 09:32:30,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [3299bae4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21181dfa, log id: 58ee0f0c >2012-12-17 09:32:31,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:31,748 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:32:31,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2695aa2a >2012-12-17 09:32:31,815 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [3299bae4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:32:32,210 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [1c20be04] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:32:32,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 739259fd >2012-12-17 09:32:32,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4c33f3bd >2012-12-17 09:32:32,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:32,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, SpmStopVDSCommand, log id: 4c33f3bd >2012-12-17 09:32:32,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, ResetIrsVDSCommand, log id: 739259fd >2012-12-17 09:32:32,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66f45f6d >2012-12-17 09:32:32,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1c20be04] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:32,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [1c20be04] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66f45f6d >2012-12-17 09:32:32,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:32,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4886 >2012-12-17 09:32:32,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4886, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bd1b2cc >2012-12-17 09:32:32,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling started: taskId = 461bae4f-8e19-444d-8a04-60e4e10b8565 >2012-12-17 09:32:33,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling ended: taskId = 461bae4f-8e19-444d-8a04-60e4e10b8565 task status = finished >2012-12-17 09:32:33,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] spmStart polling ended, spm status: SPM >2012-12-17 09:32:33,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=461bae4f-8e19-444d-8a04-60e4e10b8565), log id: 3185d6e4 >2012-12-17 09:32:33,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMClearTaskVDSCommand, log id: 3185d6e4 >2012-12-17 09:32:33,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@49df2c22, log id: 1bd1b2cc >2012-12-17 09:32:33,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:33,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:33,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11adbdc5 >2012-12-17 09:32:33,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11adbdc5 >2012-12-17 09:32:33,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2695aa2a >2012-12-17 09:32:33,946 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:32:33,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cf4e495 >2012-12-17 09:32:33,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:33,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6251276c >2012-12-17 09:32:33,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6251276c >2012-12-17 09:32:33,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [4f4f0222] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cf4e495 >2012-12-17 09:32:33,966 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [4f4f0222] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:32:35,010 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:32:42,825 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [13d7977] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:32:42,926 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [4ea30a59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:32:43,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4131f693 >2012-12-17 09:32:43,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64ba3c8e >2012-12-17 09:32:43,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:43,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStopVDSCommand, log id: 64ba3c8e >2012-12-17 09:32:43,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, ResetIrsVDSCommand, log id: 4131f693 >2012-12-17 09:32:43,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10111302 >2012-12-17 09:32:43,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:43,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10111302 >2012-12-17 09:32:45,226 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:32:53,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:53,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4887 >2012-12-17 09:32:53,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4887, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bef30a >2012-12-17 09:32:53,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling started: taskId = 64cae0d6-9741-4e03-97a0-fdd2d89de04c >2012-12-17 09:32:54,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling ended: taskId = 64cae0d6-9741-4e03-97a0-fdd2d89de04c task status = finished >2012-12-17 09:32:54,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] spmStart polling ended, spm status: SPM >2012-12-17 09:32:54,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=64cae0d6-9741-4e03-97a0-fdd2d89de04c), log id: 5efc183d >2012-12-17 09:32:54,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, HSMClearTaskVDSCommand, log id: 5efc183d >2012-12-17 09:32:54,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [6e561ed4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@789b7d82, log id: 6bef30a >2012-12-17 09:32:54,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:54,532 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:32:54,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6eed47f6 >2012-12-17 09:32:54,562 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6e561ed4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:32:54,739 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [5bc13aa6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:32:54,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ab6a0da >2012-12-17 09:32:54,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: a1d8b11 >2012-12-17 09:32:54,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:55,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, SpmStopVDSCommand, log id: a1d8b11 >2012-12-17 09:32:55,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, ResetIrsVDSCommand, log id: 2ab6a0da >2012-12-17 09:32:55,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 597c072c >2012-12-17 09:32:55,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [5bc13aa6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:55,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [5bc13aa6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 597c072c >2012-12-17 09:32:55,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:32:55,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4888 >2012-12-17 09:32:55,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4888, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6934a32 >2012-12-17 09:32:55,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling started: taskId = 1dc37269-2d6b-4e27-95a6-5a08bab7e191 >2012-12-17 09:32:56,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling ended: taskId = 1dc37269-2d6b-4e27-95a6-5a08bab7e191 task status = finished >2012-12-17 09:32:56,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] spmStart polling ended, spm status: SPM >2012-12-17 09:32:56,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dc37269-2d6b-4e27-95a6-5a08bab7e191), log id: f30b73b >2012-12-17 09:32:56,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMClearTaskVDSCommand, log id: f30b73b >2012-12-17 09:32:56,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@474d8a47, log id: 6934a32 >2012-12-17 09:32:56,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:32:56,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:56,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b53ce38 >2012-12-17 09:32:56,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b53ce38 >2012-12-17 09:32:56,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6eed47f6 >2012-12-17 09:32:56,394 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:32:56,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b07e52a >2012-12-17 09:32:56,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:32:56,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65bc4240 >2012-12-17 09:32:56,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65bc4240 >2012-12-17 09:32:56,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [651c3e82] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b07e52a >2012-12-17 09:32:56,415 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [651c3e82] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:33:05,189 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [15f75f8d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:33:05,306 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [519a0bb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:33:05,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7766e646 >2012-12-17 09:33:05,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 104469f8 >2012-12-17 09:33:05,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:06,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, SpmStopVDSCommand, log id: 104469f8 >2012-12-17 09:33:06,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, ResetIrsVDSCommand, log id: 7766e646 >2012-12-17 09:33:06,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6647da41 >2012-12-17 09:33:06,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:06,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6647da41 >2012-12-17 09:33:06,677 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:33:16,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:33:16,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4889 >2012-12-17 09:33:16,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4889, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c485ad9 >2012-12-17 09:33:16,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling started: taskId = 3e13efc3-22e9-45fa-9907-ca4183ba9c33 >2012-12-17 09:33:18,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling ended: taskId = 3e13efc3-22e9-45fa-9907-ca4183ba9c33 task status = finished >2012-12-17 09:33:18,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] spmStart polling ended, spm status: SPM >2012-12-17 09:33:18,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3e13efc3-22e9-45fa-9907-ca4183ba9c33), log id: 5640d60 >2012-12-17 09:33:18,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, HSMClearTaskVDSCommand, log id: 5640d60 >2012-12-17 09:33:18,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [3cb7ad26] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1dac3d52, log id: 4c485ad9 >2012-12-17 09:33:18,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:33:18,106 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:33:18,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2281b1de >2012-12-17 09:33:18,120 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3cb7ad26] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:33:18,219 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [632afe2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:33:18,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 655fa7db >2012-12-17 09:33:18,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 20fffb91 >2012-12-17 09:33:18,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:18,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, SpmStopVDSCommand, log id: 20fffb91 >2012-12-17 09:33:18,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, ResetIrsVDSCommand, log id: 655fa7db >2012-12-17 09:33:18,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [632afe2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 123b8401 >2012-12-17 09:33:18,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [632afe2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:18,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [632afe2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 123b8401 >2012-12-17 09:33:18,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:33:18,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4890 >2012-12-17 09:33:18,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4890, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33359f37 >2012-12-17 09:33:18,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling started: taskId = f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d >2012-12-17 09:33:19,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended: taskId = f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d task status = finished >2012-12-17 09:33:19,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended, spm status: SPM >2012-12-17 09:33:19,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3ff19a0-e5f2-41b2-9bc5-5ea5cc3e0f0d), log id: 11b5bbfd >2012-12-17 09:33:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMClearTaskVDSCommand, log id: 11b5bbfd >2012-12-17 09:33:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3248e820, log id: 33359f37 >2012-12-17 09:33:19,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:33:19,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:19,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 719c5552 >2012-12-17 09:33:19,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 719c5552 >2012-12-17 09:33:19,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2281b1de >2012-12-17 09:33:19,830 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:33:19,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ea5b4c3 >2012-12-17 09:33:19,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:19,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f2dbc14 >2012-12-17 09:33:19,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f2dbc14 >2012-12-17 09:33:19,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4ea5b4c3 >2012-12-17 09:33:19,852 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [263a433b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:33:27,246 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:33:28,662 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d7157bb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:33:28,761 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [20d09da7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:33:28,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6862fda1 >2012-12-17 09:33:28,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7df27e52 >2012-12-17 09:33:28,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:29,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SpmStopVDSCommand, log id: 7df27e52 >2012-12-17 09:33:29,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, ResetIrsVDSCommand, log id: 6862fda1 >2012-12-17 09:33:29,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7aecc102 >2012-12-17 09:33:29,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:29,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7aecc102 >2012-12-17 09:33:37,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:33:39,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:33:39,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4891 >2012-12-17 09:33:39,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4891, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71cdfe23 >2012-12-17 09:33:39,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling started: taskId = 14253e40-f1a1-4353-95b4-8c38536e0571 >2012-12-17 09:33:40,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended: taskId = 14253e40-f1a1-4353-95b4-8c38536e0571 task status = finished >2012-12-17 09:33:40,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] spmStart polling ended, spm status: SPM >2012-12-17 09:33:40,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14253e40-f1a1-4353-95b4-8c38536e0571), log id: 2f86007c >2012-12-17 09:33:40,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, HSMClearTaskVDSCommand, log id: 2f86007c >2012-12-17 09:33:40,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [57cdd675] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@576adb89, log id: 71cdfe23 >2012-12-17 09:33:40,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:33:41,025 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:33:41,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72342b75 >2012-12-17 09:33:41,057 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [57cdd675] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:33:41,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [680acadd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:33:41,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ccd3485 >2012-12-17 09:33:41,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13786ef0 >2012-12-17 09:33:41,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:41,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, SpmStopVDSCommand, log id: 13786ef0 >2012-12-17 09:33:41,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, ResetIrsVDSCommand, log id: 3ccd3485 >2012-12-17 09:33:41,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [680acadd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 114156fd >2012-12-17 09:33:41,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:41,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [680acadd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 114156fd >2012-12-17 09:33:41,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:33:41,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4892 >2012-12-17 09:33:41,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4892, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bf76416 >2012-12-17 09:33:41,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = ab93a84a-8847-4966-9c0a-5828b2b4835c >2012-12-17 09:33:42,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = ab93a84a-8847-4966-9c0a-5828b2b4835c task status = finished >2012-12-17 09:33:42,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM >2012-12-17 09:33:42,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab93a84a-8847-4966-9c0a-5828b2b4835c), log id: 6d5cddae >2012-12-17 09:33:42,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 6d5cddae >2012-12-17 09:33:42,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6bde0299, log id: bf76416 >2012-12-17 09:33:42,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:33:42,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:42,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1b89b64a >2012-12-17 09:33:42,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b89b64a >2012-12-17 09:33:42,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72342b75 >2012-12-17 09:33:42,744 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:33:42,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e1ec265 >2012-12-17 09:33:42,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:42,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6c0c8117 >2012-12-17 09:33:42,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c0c8117 >2012-12-17 09:33:42,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e1ec265 >2012-12-17 09:33:42,764 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:33:47,657 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:33:51,624 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [68b3db6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:33:51,791 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [56a7c4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:33:51,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7b975bfc >2012-12-17 09:33:51,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 458c2f56 >2012-12-17 09:33:51,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:52,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStopVDSCommand, log id: 458c2f56 >2012-12-17 09:33:52,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, ResetIrsVDSCommand, log id: 7b975bfc >2012-12-17 09:33:52,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44d3ce62 >2012-12-17 09:33:52,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:33:52,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44d3ce62 >2012-12-17 09:33:58,253 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:34:02,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:02,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4893 >2012-12-17 09:34:02,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4893, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3decbd81 >2012-12-17 09:34:02,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling started: taskId = 4f2185bc-a034-4630-8db5-de6a45452d80 >2012-12-17 09:34:03,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling ended: taskId = 4f2185bc-a034-4630-8db5-de6a45452d80 task status = finished >2012-12-17 09:34:03,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] spmStart polling ended, spm status: SPM >2012-12-17 09:34:03,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4f2185bc-a034-4630-8db5-de6a45452d80), log id: 5d7f6966 >2012-12-17 09:34:03,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, HSMClearTaskVDSCommand, log id: 5d7f6966 >2012-12-17 09:34:03,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [3f9576fc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57e0d77e, log id: 3decbd81 >2012-12-17 09:34:03,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:03,394 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:34:03,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2bbd5a7c >2012-12-17 09:34:03,405 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [3f9576fc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:34:03,814 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [248935dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:34:03,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28344361 >2012-12-17 09:34:03,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2348d3b0 >2012-12-17 09:34:03,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:04,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, SpmStopVDSCommand, log id: 2348d3b0 >2012-12-17 09:34:04,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, ResetIrsVDSCommand, log id: 28344361 >2012-12-17 09:34:04,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53d6a1be >2012-12-17 09:34:04,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:04,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53d6a1be >2012-12-17 09:34:04,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:04,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4894 >2012-12-17 09:34:04,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4894, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fc15b4e >2012-12-17 09:34:04,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling started: taskId = 2d6245f2-535f-4bef-9fe2-f63187688762 >2012-12-17 09:34:05,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling ended: taskId = 2d6245f2-535f-4bef-9fe2-f63187688762 task status = finished >2012-12-17 09:34:05,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] spmStart polling ended, spm status: SPM >2012-12-17 09:34:05,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2d6245f2-535f-4bef-9fe2-f63187688762), log id: 395a1f87 >2012-12-17 09:34:05,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMClearTaskVDSCommand, log id: 395a1f87 >2012-12-17 09:34:05,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4bcbf57b, log id: 3fc15b4e >2012-12-17 09:34:05,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:05,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c8604f9 >2012-12-17 09:34:05,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c8604f9 >2012-12-17 09:34:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2bbd5a7c >2012-12-17 09:34:05,363 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:34:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b0967a >2012-12-17 09:34:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:05,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 598c391a >2012-12-17 09:34:05,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 598c391a >2012-12-17 09:34:05,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [22fa8e65] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b0967a >2012-12-17 09:34:05,383 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [22fa8e65] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:34:08,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:34:14,260 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [20c0de80] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:34:14,464 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [3c4cd577] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:34:14,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36fd4c2f >2012-12-17 09:34:14,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21e7e53f >2012-12-17 09:34:14,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:14,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, SpmStopVDSCommand, log id: 21e7e53f >2012-12-17 09:34:14,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, ResetIrsVDSCommand, log id: 36fd4c2f >2012-12-17 09:34:14,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5daa386a >2012-12-17 09:34:14,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:14,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5daa386a >2012-12-17 09:34:18,666 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:34:24,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:24,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4895 >2012-12-17 09:34:24,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4895, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c40109b >2012-12-17 09:34:24,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling started: taskId = 5d7fbd7e-29e2-4420-89db-ff1ad972f0ea >2012-12-17 09:34:25,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended: taskId = 5d7fbd7e-29e2-4420-89db-ff1ad972f0ea task status = finished >2012-12-17 09:34:25,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] spmStart polling ended, spm status: SPM >2012-12-17 09:34:25,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5d7fbd7e-29e2-4420-89db-ff1ad972f0ea), log id: 4f3ba92 >2012-12-17 09:34:25,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, HSMClearTaskVDSCommand, log id: 4f3ba92 >2012-12-17 09:34:25,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [75b7a65b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d5dc9c8, log id: 2c40109b >2012-12-17 09:34:26,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:26,291 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:34:26,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fa88926 >2012-12-17 09:34:26,313 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [75b7a65b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:34:26,497 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [47513fa0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:34:26,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40dde40 >2012-12-17 09:34:26,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 307b6232 >2012-12-17 09:34:26,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:26,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SpmStopVDSCommand, log id: 307b6232 >2012-12-17 09:34:26,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, ResetIrsVDSCommand, log id: 40dde40 >2012-12-17 09:34:26,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5304d3ef >2012-12-17 09:34:26,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:26,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5304d3ef >2012-12-17 09:34:27,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:27,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4896 >2012-12-17 09:34:27,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4896, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cea505f >2012-12-17 09:34:27,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7 >2012-12-17 09:34:28,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7 task status = finished >2012-12-17 09:34:28,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM >2012-12-17 09:34:28,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7741f8e-a3cc-4230-9239-5e3dcf1b4ab7), log id: ce92b62 >2012-12-17 09:34:28,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: ce92b62 >2012-12-17 09:34:28,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@201f358d, log id: 6cea505f >2012-12-17 09:34:28,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:28,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:28,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3b868cd8 >2012-12-17 09:34:28,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b868cd8 >2012-12-17 09:34:28,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fa88926 >2012-12-17 09:34:28,187 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:34:28,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32b2652d >2012-12-17 09:34:28,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:28,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5baf1a4d >2012-12-17 09:34:28,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5baf1a4d >2012-12-17 09:34:28,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32b2652d >2012-12-17 09:34:28,208 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:34:28,847 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:34:38,355 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5f63eb29] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:34:38,692 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [5117cf2d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:34:38,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7f8d3f70 >2012-12-17 09:34:38,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46ecaa03 >2012-12-17 09:34:38,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:38,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SpmStopVDSCommand, log id: 46ecaa03 >2012-12-17 09:34:38,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, ResetIrsVDSCommand, log id: 7f8d3f70 >2012-12-17 09:34:38,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7593f22a >2012-12-17 09:34:38,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:38,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7593f22a >2012-12-17 09:34:41,285 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:34:49,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:49,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4897 >2012-12-17 09:34:49,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4897, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e983992 >2012-12-17 09:34:49,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling started: taskId = 86c07b01-3a81-477f-84bd-8e0559cb534b >2012-12-17 09:34:50,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling ended: taskId = 86c07b01-3a81-477f-84bd-8e0559cb534b task status = finished >2012-12-17 09:34:50,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] spmStart polling ended, spm status: SPM >2012-12-17 09:34:50,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=86c07b01-3a81-477f-84bd-8e0559cb534b), log id: 37931e7c >2012-12-17 09:34:50,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, HSMClearTaskVDSCommand, log id: 37931e7c >2012-12-17 09:34:50,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [6de308ea] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bb67b65, log id: 3e983992 >2012-12-17 09:34:50,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:50,177 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:34:50,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2cf64ece >2012-12-17 09:34:50,197 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6de308ea] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:34:50,294 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [6728cccd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:34:50,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48888843 >2012-12-17 09:34:50,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4274287 >2012-12-17 09:34:50,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:50,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, SpmStopVDSCommand, log id: 4274287 >2012-12-17 09:34:51,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, ResetIrsVDSCommand, log id: 48888843 >2012-12-17 09:34:51,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f35b6cd >2012-12-17 09:34:51,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6728cccd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:51,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6728cccd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f35b6cd >2012-12-17 09:34:51,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:34:51,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4898 >2012-12-17 09:34:51,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4898, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 504f3eae >2012-12-17 09:34:51,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = da3e029a-8e98-4c5c-8b6b-cd39190c533d >2012-12-17 09:34:52,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = da3e029a-8e98-4c5c-8b6b-cd39190c533d task status = finished >2012-12-17 09:34:52,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM >2012-12-17 09:34:52,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=da3e029a-8e98-4c5c-8b6b-cd39190c533d), log id: 579277e3 >2012-12-17 09:34:52,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 579277e3 >2012-12-17 09:34:52,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50fd04ff, log id: 504f3eae >2012-12-17 09:34:52,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:34:52,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:52,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2055255 >2012-12-17 09:34:52,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2055255 >2012-12-17 09:34:52,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2cf64ece >2012-12-17 09:34:52,538 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:34:52,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3efbd023 >2012-12-17 09:34:52,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:34:52,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6694de54 >2012-12-17 09:34:52,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6694de54 >2012-12-17 09:34:52,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3efbd023 >2012-12-17 09:34:52,558 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Checking autorecoverable hosts >2012-12-17 09:35:00,012 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Autorecovering 0 hosts >2012-12-17 09:35:00,013 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Checking autorecoverable hosts done >2012-12-17 09:35:00,013 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [6e84ec96] Autorecovering storage domains is disabled, skipping >2012-12-17 09:35:01,464 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [e42ef5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:01,608 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [4f008eee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:01,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c694e10 >2012-12-17 09:35:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: adb73d0 >2012-12-17 09:35:01,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, SpmStopVDSCommand, log id: adb73d0 >2012-12-17 09:35:01,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, ResetIrsVDSCommand, log id: 6c694e10 >2012-12-17 09:35:01,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b031d7e >2012-12-17 09:35:01,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:01,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b031d7e >2012-12-17 09:35:02,041 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:35:11,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:12,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4899 >2012-12-17 09:35:12,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4899, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 297ebee2 >2012-12-17 09:35:12,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling started: taskId = c6121edf-5dc4-49c8-8161-87a076b4c9ad >2012-12-17 09:35:13,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended: taskId = c6121edf-5dc4-49c8-8161-87a076b4c9ad task status = finished >2012-12-17 09:35:13,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] spmStart polling ended, spm status: SPM >2012-12-17 09:35:13,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6121edf-5dc4-49c8-8161-87a076b4c9ad), log id: 3950effd >2012-12-17 09:35:13,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, HSMClearTaskVDSCommand, log id: 3950effd >2012-12-17 09:35:13,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [3aea96b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70bbe870, log id: 297ebee2 >2012-12-17 09:35:13,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:35:13,252 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:35:13,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1aa90823 >2012-12-17 09:35:13,274 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3aea96b8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:13,392 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [24a780e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:13,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1ea6a0e8 >2012-12-17 09:35:13,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3595ca6 >2012-12-17 09:35:13,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:13,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, SpmStopVDSCommand, log id: 3595ca6 >2012-12-17 09:35:13,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, ResetIrsVDSCommand, log id: 1ea6a0e8 >2012-12-17 09:35:13,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36c7e5f0 >2012-12-17 09:35:13,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [24a780e1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:13,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [24a780e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36c7e5f0 >2012-12-17 09:35:13,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:13,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4900 >2012-12-17 09:35:13,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4900, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ee5cd08 >2012-12-17 09:35:13,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling started: taskId = 8a34f209-108f-4245-9cce-78366d415c39 >2012-12-17 09:35:14,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling ended: taskId = 8a34f209-108f-4245-9cce-78366d415c39 task status = finished >2012-12-17 09:35:14,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] spmStart polling ended, spm status: SPM >2012-12-17 09:35:14,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a34f209-108f-4245-9cce-78366d415c39), log id: 5a7eb679 >2012-12-17 09:35:14,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMClearTaskVDSCommand, log id: 5a7eb679 >2012-12-17 09:35:14,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34fdf059, log id: 4ee5cd08 >2012-12-17 09:35:14,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:35:14,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:14,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 8407678 >2012-12-17 09:35:14,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 8407678 >2012-12-17 09:35:14,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1aa90823 >2012-12-17 09:35:14,857 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:35:14,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68ec9559 >2012-12-17 09:35:14,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:14,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4311d6a1 >2012-12-17 09:35:14,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4311d6a1 >2012-12-17 09:35:14,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-33) [47513fa0] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68ec9559 >2012-12-17 09:35:14,877 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) [47513fa0] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:35:22,399 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:35:23,722 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [26c9f399] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:23,805 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [588a5ec3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:23,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5a77ac42 >2012-12-17 09:35:23,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b89cd7b >2012-12-17 09:35:23,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:24,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, SpmStopVDSCommand, log id: 6b89cd7b >2012-12-17 09:35:24,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, ResetIrsVDSCommand, log id: 5a77ac42 >2012-12-17 09:35:24,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c897660 >2012-12-17 09:35:24,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:24,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c897660 >2012-12-17 09:35:32,573 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:35:34,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:34,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4901 >2012-12-17 09:35:34,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4901, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5595c69a >2012-12-17 09:35:34,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling started: taskId = 0c5ffeea-aa83-479f-850d-8636d2e6bc5a >2012-12-17 09:35:36,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling ended: taskId = 0c5ffeea-aa83-479f-850d-8636d2e6bc5a task status = finished >2012-12-17 09:35:36,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] spmStart polling ended, spm status: SPM >2012-12-17 09:35:36,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0c5ffeea-aa83-479f-850d-8636d2e6bc5a), log id: 39e22398 >2012-12-17 09:35:36,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, HSMClearTaskVDSCommand, log id: 39e22398 >2012-12-17 09:35:36,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6b59f114] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d8eedd8, log id: 5595c69a >2012-12-17 09:35:36,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:35:36,385 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:35:36,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c57853c >2012-12-17 09:35:36,433 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6b59f114] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:36,626 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [3623f38c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:36,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79a88e45 >2012-12-17 09:35:36,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11174adf >2012-12-17 09:35:36,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:37,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, SpmStopVDSCommand, log id: 11174adf >2012-12-17 09:35:37,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, ResetIrsVDSCommand, log id: 79a88e45 >2012-12-17 09:35:37,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e63c62e >2012-12-17 09:35:37,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3623f38c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:37,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [3623f38c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e63c62e >2012-12-17 09:35:37,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:37,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4902 >2012-12-17 09:35:37,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4902, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 106db34b >2012-12-17 09:35:37,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling started: taskId = 37554cd1-fddc-4a0f-abba-b722833109f8 >2012-12-17 09:35:38,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended: taskId = 37554cd1-fddc-4a0f-abba-b722833109f8 task status = finished >2012-12-17 09:35:38,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] spmStart polling ended, spm status: SPM >2012-12-17 09:35:38,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=37554cd1-fddc-4a0f-abba-b722833109f8), log id: 65e9ae6a >2012-12-17 09:35:38,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMClearTaskVDSCommand, log id: 65e9ae6a >2012-12-17 09:35:38,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50b9e56b, log id: 106db34b >2012-12-17 09:35:38,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:35:38,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:38,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 32a2a052 >2012-12-17 09:35:38,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a2a052 >2012-12-17 09:35:38,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c57853c >2012-12-17 09:35:38,505 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:35:38,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52cb8176 >2012-12-17 09:35:38,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:38,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e20938f >2012-12-17 09:35:38,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e20938f >2012-12-17 09:35:38,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-80) [6c899ac8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52cb8176 >2012-12-17 09:35:38,525 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) [6c899ac8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:35:42,802 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:35:47,372 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [1813139a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:47,466 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [134cc8af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:47,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e35d1b >2012-12-17 09:35:47,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a65acc3 >2012-12-17 09:35:47,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:47,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, SpmStopVDSCommand, log id: 4a65acc3 >2012-12-17 09:35:47,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, ResetIrsVDSCommand, log id: 3e35d1b >2012-12-17 09:35:47,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41f155f5 >2012-12-17 09:35:47,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:47,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41f155f5 >2012-12-17 09:35:52,984 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:35:57,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:57,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4903 >2012-12-17 09:35:57,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4903, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 27f340d7 >2012-12-17 09:35:57,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling started: taskId = 1fa20210-8269-4a07-bed2-a716fd5ea40d >2012-12-17 09:35:58,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling ended: taskId = 1fa20210-8269-4a07-bed2-a716fd5ea40d task status = finished >2012-12-17 09:35:58,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] spmStart polling ended, spm status: SPM >2012-12-17 09:35:58,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [23d69744] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1fa20210-8269-4a07-bed2-a716fd5ea40d), log id: 54e5eb19 >2012-12-17 09:35:58,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, HSMClearTaskVDSCommand, log id: 54e5eb19 >2012-12-17 09:35:58,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [23d69744] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42ade1da, log id: 27f340d7 >2012-12-17 09:35:58,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:35:59,018 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:35:59,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 361355fc >2012-12-17 09:35:59,031 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [23d69744] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:35:59,184 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:35:59,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5eaaedb4 >2012-12-17 09:35:59,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ae77271 >2012-12-17 09:35:59,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:59,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, SpmStopVDSCommand, log id: 5ae77271 >2012-12-17 09:35:59,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, ResetIrsVDSCommand, log id: 5eaaedb4 >2012-12-17 09:35:59,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 521d9b6d >2012-12-17 09:35:59,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:35:59,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 521d9b6d >2012-12-17 09:35:59,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:35:59,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4904 >2012-12-17 09:35:59,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4904, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 413510a2 >2012-12-17 09:35:59,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling started: taskId = bf526500-b163-4b84-8888-728a0730dc8a >2012-12-17 09:36:00,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling ended: taskId = bf526500-b163-4b84-8888-728a0730dc8a task status = finished >2012-12-17 09:36:00,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] spmStart polling ended, spm status: SPM >2012-12-17 09:36:00,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bf526500-b163-4b84-8888-728a0730dc8a), log id: c38863f >2012-12-17 09:36:00,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMClearTaskVDSCommand, log id: c38863f >2012-12-17 09:36:00,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52f5985d, log id: 413510a2 >2012-12-17 09:36:01,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:36:01,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:01,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: c4ef89e >2012-12-17 09:36:01,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: c4ef89e >2012-12-17 09:36:01,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 361355fc >2012-12-17 09:36:01,203 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:01,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14a45362 >2012-12-17 09:36:01,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:01,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4033a5c9 >2012-12-17 09:36:01,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4033a5c9 >2012-12-17 09:36:01,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-24) [6157bb5e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14a45362 >2012-12-17 09:36:01,223 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-24) [6157bb5e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:03,191 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:36:09,531 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3fe36f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:36:09,657 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [46833f1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:36:09,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4a9d735f >2012-12-17 09:36:09,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d1785f >2012-12-17 09:36:09,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:09,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, SpmStopVDSCommand, log id: 7d1785f >2012-12-17 09:36:09,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, ResetIrsVDSCommand, log id: 4a9d735f >2012-12-17 09:36:09,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c76af09 >2012-12-17 09:36:09,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:09,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c76af09 >2012-12-17 09:36:13,367 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:36:19,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:36:20,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4905 >2012-12-17 09:36:20,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4905, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ddece82 >2012-12-17 09:36:20,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling started: taskId = b55918ca-9ce6-44f7-adc0-15082b41077e >2012-12-17 09:36:21,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended: taskId = b55918ca-9ce6-44f7-adc0-15082b41077e task status = finished >2012-12-17 09:36:21,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] spmStart polling ended, spm status: SPM >2012-12-17 09:36:21,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b55918ca-9ce6-44f7-adc0-15082b41077e), log id: 5a0eded5 >2012-12-17 09:36:21,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, HSMClearTaskVDSCommand, log id: 5a0eded5 >2012-12-17 09:36:21,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@647a024, log id: 6ddece82 >2012-12-17 09:36:21,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:36:22,030 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:36:22,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6bde588 >2012-12-17 09:36:22,079 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7a1c9fc6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:36:22,284 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:36:22,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39bc936 >2012-12-17 09:36:22,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1b1b73ac >2012-12-17 09:36:22,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:22,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, SpmStopVDSCommand, log id: 1b1b73ac >2012-12-17 09:36:22,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, ResetIrsVDSCommand, log id: 39bc936 >2012-12-17 09:36:22,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e1049a >2012-12-17 09:36:22,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:22,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e1049a >2012-12-17 09:36:22,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:36:22,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4906 >2012-12-17 09:36:22,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4906, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 515c06da >2012-12-17 09:36:22,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling started: taskId = a6eacabb-6e2b-477f-95de-3df25a85d5e1 >2012-12-17 09:36:23,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended: taskId = a6eacabb-6e2b-477f-95de-3df25a85d5e1 task status = finished >2012-12-17 09:36:23,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] spmStart polling ended, spm status: SPM >2012-12-17 09:36:23,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a6eacabb-6e2b-477f-95de-3df25a85d5e1), log id: 1a1e6aaf >2012-12-17 09:36:23,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMClearTaskVDSCommand, log id: 1a1e6aaf >2012-12-17 09:36:23,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42ee148e, log id: 515c06da >2012-12-17 09:36:23,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:36:23,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:23,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e8be767 >2012-12-17 09:36:23,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e8be767 >2012-12-17 09:36:23,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6bde588 >2012-12-17 09:36:23,679 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:23,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e9120f6 >2012-12-17 09:36:23,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:23,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2867c319 >2012-12-17 09:36:23,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2867c319 >2012-12-17 09:36:23,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [3a1c2706] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e9120f6 >2012-12-17 09:36:23,699 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [3a1c2706] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:32,589 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [783fb511] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:36:32,738 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [9b1d86] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:36:32,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4150d8c2 >2012-12-17 09:36:32,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 757266b8 >2012-12-17 09:36:32,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:33,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, SpmStopVDSCommand, log id: 757266b8 >2012-12-17 09:36:33,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, ResetIrsVDSCommand, log id: 4150d8c2 >2012-12-17 09:36:33,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14fbde2 >2012-12-17 09:36:33,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:33,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14fbde2 >2012-12-17 09:36:33,738 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:36:43,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:36:43,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4907 >2012-12-17 09:36:43,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4907, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58c1ba2f >2012-12-17 09:36:43,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling started: taskId = a021c2db-fa67-4743-8467-9eb36e724145 >2012-12-17 09:36:44,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling ended: taskId = a021c2db-fa67-4743-8467-9eb36e724145 task status = finished >2012-12-17 09:36:44,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] spmStart polling ended, spm status: SPM >2012-12-17 09:36:44,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a021c2db-fa67-4743-8467-9eb36e724145), log id: 77b95030 >2012-12-17 09:36:44,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, HSMClearTaskVDSCommand, log id: 77b95030 >2012-12-17 09:36:44,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [47111e6d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@273fbcb8, log id: 58c1ba2f >2012-12-17 09:36:44,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:36:44,577 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:36:44,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c5701b7 >2012-12-17 09:36:44,602 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [47111e6d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:36:44,765 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [515f648d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:36:44,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ea5558a >2012-12-17 09:36:44,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 69f7940b >2012-12-17 09:36:44,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:44,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SpmStopVDSCommand, log id: 69f7940b >2012-12-17 09:36:44,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, ResetIrsVDSCommand, log id: 5ea5558a >2012-12-17 09:36:45,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44bcbd15 >2012-12-17 09:36:45,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:45,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44bcbd15 >2012-12-17 09:36:45,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:36:45,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4908 >2012-12-17 09:36:45,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4908, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4cbb6160 >2012-12-17 09:36:45,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling started: taskId = 3c84b1d1-004b-4f00-b887-1575c8e1ac6a >2012-12-17 09:36:46,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling ended: taskId = 3c84b1d1-004b-4f00-b887-1575c8e1ac6a task status = finished >2012-12-17 09:36:46,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] spmStart polling ended, spm status: SPM >2012-12-17 09:36:46,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3c84b1d1-004b-4f00-b887-1575c8e1ac6a), log id: 2ffc14d3 >2012-12-17 09:36:46,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMClearTaskVDSCommand, log id: 2ffc14d3 >2012-12-17 09:36:46,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ebb4141, log id: 4cbb6160 >2012-12-17 09:36:46,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:36:46,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:46,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f18529e >2012-12-17 09:36:46,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f18529e >2012-12-17 09:36:46,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1c5701b7 >2012-12-17 09:36:46,256 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:46,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55d8236e >2012-12-17 09:36:46,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:46,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: e673a8c >2012-12-17 09:36:46,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: e673a8c >2012-12-17 09:36:46,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [7db92791] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 55d8236e >2012-12-17 09:36:46,277 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [7db92791] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:36:54,151 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:36:55,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [651c3e82] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:36:55,234 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [681f9614] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:36:55,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 195798c7 >2012-12-17 09:36:55,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42bf564f >2012-12-17 09:36:55,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:55,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, SpmStopVDSCommand, log id: 42bf564f >2012-12-17 09:36:55,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, ResetIrsVDSCommand, log id: 195798c7 >2012-12-17 09:36:55,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [681f9614] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45c354e2 >2012-12-17 09:36:55,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [681f9614] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:36:55,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [681f9614] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45c354e2 >2012-12-17 09:37:04,333 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:37:06,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:06,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4909 >2012-12-17 09:37:06,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4909, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77e97861 >2012-12-17 09:37:06,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling started: taskId = 1b4541eb-d7a6-498a-a8bf-076a05734411 >2012-12-17 09:37:07,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling ended: taskId = 1b4541eb-d7a6-498a-a8bf-076a05734411 task status = finished >2012-12-17 09:37:07,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] spmStart polling ended, spm status: SPM >2012-12-17 09:37:07,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [570659e5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1b4541eb-d7a6-498a-a8bf-076a05734411), log id: 21be2b44 >2012-12-17 09:37:07,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, HSMClearTaskVDSCommand, log id: 21be2b44 >2012-12-17 09:37:07,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [570659e5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c0a8eaa, log id: 77e97861 >2012-12-17 09:37:07,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:07,867 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:37:07,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1ae91126 >2012-12-17 09:37:07,891 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [570659e5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:37:08,003 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [7646beaa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:37:08,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6da4d875 >2012-12-17 09:37:08,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7fdcabe3 >2012-12-17 09:37:08,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:08,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStopVDSCommand, log id: 7fdcabe3 >2012-12-17 09:37:08,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, ResetIrsVDSCommand, log id: 6da4d875 >2012-12-17 09:37:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 218cc4b1 >2012-12-17 09:37:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:08,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 218cc4b1 >2012-12-17 09:37:08,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:08,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4910 >2012-12-17 09:37:08,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4910, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33afb4a4 >2012-12-17 09:37:08,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = f090005a-716f-4c55-a81d-0bf7e3e2c3f4 >2012-12-17 09:37:09,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = f090005a-716f-4c55-a81d-0bf7e3e2c3f4 task status = finished >2012-12-17 09:37:09,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM >2012-12-17 09:37:09,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f090005a-716f-4c55-a81d-0bf7e3e2c3f4), log id: 26c8ea02 >2012-12-17 09:37:09,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 26c8ea02 >2012-12-17 09:37:09,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@507b3d32, log id: 33afb4a4 >2012-12-17 09:37:09,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:09,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:09,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 47d52282 >2012-12-17 09:37:09,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47d52282 >2012-12-17 09:37:09,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1ae91126 >2012-12-17 09:37:09,440 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:09,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79cca9a5 >2012-12-17 09:37:09,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:09,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ebc8b16 >2012-12-17 09:37:09,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ebc8b16 >2012-12-17 09:37:09,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79cca9a5 >2012-12-17 09:37:09,461 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:14,519 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:37:18,314 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1c20be04] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:37:18,450 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [6eb2e264] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:37:18,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 663fa7c >2012-12-17 09:37:18,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 11b77f47 >2012-12-17 09:37:18,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:18,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStopVDSCommand, log id: 11b77f47 >2012-12-17 09:37:18,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, ResetIrsVDSCommand, log id: 663fa7c >2012-12-17 09:37:18,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67adc956 >2012-12-17 09:37:18,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:18,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67adc956 >2012-12-17 09:37:24,711 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:37:28,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:28,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4911 >2012-12-17 09:37:28,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4911, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 392dad84 >2012-12-17 09:37:28,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling started: taskId = ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec >2012-12-17 09:37:29,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended: taskId = ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec task status = finished >2012-12-17 09:37:29,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] spmStart polling ended, spm status: SPM >2012-12-17 09:37:29,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae0f46a6-da6f-4d41-b1a1-90ba3b0564ec), log id: 53831a7b >2012-12-17 09:37:29,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, HSMClearTaskVDSCommand, log id: 53831a7b >2012-12-17 09:37:29,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [7f190dc3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@443fac12, log id: 392dad84 >2012-12-17 09:37:29,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:30,015 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:37:30,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cf7efd2 >2012-12-17 09:37:30,040 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7f190dc3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:37:30,149 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [2e298a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:37:30,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 402563d3 >2012-12-17 09:37:30,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d354812 >2012-12-17 09:37:30,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:30,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, SpmStopVDSCommand, log id: 6d354812 >2012-12-17 09:37:30,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, ResetIrsVDSCommand, log id: 402563d3 >2012-12-17 09:37:30,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79bd14f4 >2012-12-17 09:37:30,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:30,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79bd14f4 >2012-12-17 09:37:31,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:31,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4912 >2012-12-17 09:37:31,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4912, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b4ffea4 >2012-12-17 09:37:31,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling started: taskId = bbcc6eb4-d8ef-415f-8740-908f7a6ee040 >2012-12-17 09:37:32,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended: taskId = bbcc6eb4-d8ef-415f-8740-908f7a6ee040 task status = finished >2012-12-17 09:37:32,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended, spm status: SPM >2012-12-17 09:37:32,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bbcc6eb4-d8ef-415f-8740-908f7a6ee040), log id: 78da53c3 >2012-12-17 09:37:32,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMClearTaskVDSCommand, log id: 78da53c3 >2012-12-17 09:37:32,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45b16531, log id: 3b4ffea4 >2012-12-17 09:37:32,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:32,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:32,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5187094a >2012-12-17 09:37:32,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5187094a >2012-12-17 09:37:32,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cf7efd2 >2012-12-17 09:37:32,618 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:32,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 78730316 >2012-12-17 09:37:32,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:32,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f5ec121 >2012-12-17 09:37:32,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f5ec121 >2012-12-17 09:37:32,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78730316 >2012-12-17 09:37:32,638 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [36643dd7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:34,911 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:37:41,514 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [5bc13aa6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:37:41,655 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:37:41,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f24ac3a >2012-12-17 09:37:41,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e7f58a1 >2012-12-17 09:37:41,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:42,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, SpmStopVDSCommand, log id: 7e7f58a1 >2012-12-17 09:37:42,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, ResetIrsVDSCommand, log id: 3f24ac3a >2012-12-17 09:37:42,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 256e51a4 >2012-12-17 09:37:42,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:42,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 256e51a4 >2012-12-17 09:37:45,099 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:37:52,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:52,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4913 >2012-12-17 09:37:52,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4913, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48deaec4 >2012-12-17 09:37:52,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling started: taskId = 48a82fd4-23b7-45a1-a135-cad7b3c426c6 >2012-12-17 09:37:53,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended: taskId = 48a82fd4-23b7-45a1-a135-cad7b3c426c6 task status = finished >2012-12-17 09:37:53,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] spmStart polling ended, spm status: SPM >2012-12-17 09:37:53,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48a82fd4-23b7-45a1-a135-cad7b3c426c6), log id: 466cb3ea >2012-12-17 09:37:53,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, HSMClearTaskVDSCommand, log id: 466cb3ea >2012-12-17 09:37:53,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [160c510e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a97ebc5, log id: 48deaec4 >2012-12-17 09:37:53,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:53,326 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:37:53,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 464a9adc >2012-12-17 09:37:53,339 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [160c510e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:37:53,523 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:37:53,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b9d6763 >2012-12-17 09:37:53,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5fe02130 >2012-12-17 09:37:53,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:53,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, SpmStopVDSCommand, log id: 5fe02130 >2012-12-17 09:37:53,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, ResetIrsVDSCommand, log id: b9d6763 >2012-12-17 09:37:53,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cf80d4d >2012-12-17 09:37:53,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:53,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cf80d4d >2012-12-17 09:37:53,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:37:53,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4914 >2012-12-17 09:37:53,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4914, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68f47bb2 >2012-12-17 09:37:53,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling started: taskId = 67a50722-4cd2-4f67-afc2-77b541477239 >2012-12-17 09:37:54,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling ended: taskId = 67a50722-4cd2-4f67-afc2-77b541477239 task status = finished >2012-12-17 09:37:55,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] spmStart polling ended, spm status: SPM >2012-12-17 09:37:55,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=67a50722-4cd2-4f67-afc2-77b541477239), log id: 72a0ffea >2012-12-17 09:37:55,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMClearTaskVDSCommand, log id: 72a0ffea >2012-12-17 09:37:55,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@28019671, log id: 68f47bb2 >2012-12-17 09:37:55,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:37:55,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:55,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b1bb827 >2012-12-17 09:37:55,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b1bb827 >2012-12-17 09:37:55,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 464a9adc >2012-12-17 09:37:55,098 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:55,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 674c5ea9 >2012-12-17 09:37:55,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:37:55,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6da1ae1 >2012-12-17 09:37:55,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6da1ae1 >2012-12-17 09:37:55,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [134ba83e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 674c5ea9 >2012-12-17 09:37:55,118 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [134ba83e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:37:55,350 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:38:03,979 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [22fa8e65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:38:04,086 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:38:04,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5668e9a >2012-12-17 09:38:04,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e59458b >2012-12-17 09:38:04,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:04,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, SpmStopVDSCommand, log id: 3e59458b >2012-12-17 09:38:04,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, ResetIrsVDSCommand, log id: 5668e9a >2012-12-17 09:38:04,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 383cc203 >2012-12-17 09:38:04,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:04,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 383cc203 >2012-12-17 09:38:06,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:38:14,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:38:14,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4915 >2012-12-17 09:38:14,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4915, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42877e07 >2012-12-17 09:38:14,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling started: taskId = 7198a7bf-f832-4c02-8498-22e320d0da29 >2012-12-17 09:38:15,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling ended: taskId = 7198a7bf-f832-4c02-8498-22e320d0da29 task status = finished >2012-12-17 09:38:15,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] spmStart polling ended, spm status: SPM >2012-12-17 09:38:15,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7198a7bf-f832-4c02-8498-22e320d0da29), log id: 5bda95f1 >2012-12-17 09:38:15,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, HSMClearTaskVDSCommand, log id: 5bda95f1 >2012-12-17 09:38:15,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [3c4cd577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6baa3053, log id: 42877e07 >2012-12-17 09:38:15,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:38:16,159 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:38:16,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5957bc7a >2012-12-17 09:38:16,177 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [3c4cd577] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:38:16,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [2d67b14a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:38:16,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f4c6fb6 >2012-12-17 09:38:16,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 297af071 >2012-12-17 09:38:16,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:16,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStopVDSCommand, log id: 297af071 >2012-12-17 09:38:16,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, ResetIrsVDSCommand, log id: 6f4c6fb6 >2012-12-17 09:38:16,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f5d3871 >2012-12-17 09:38:16,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:16,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f5d3871 >2012-12-17 09:38:16,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:38:16,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4916 >2012-12-17 09:38:16,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4916, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72212314 >2012-12-17 09:38:16,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling started: taskId = e5c64db9-d242-4cde-8c96-d1d69a6317a0 >2012-12-17 09:38:17,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended: taskId = e5c64db9-d242-4cde-8c96-d1d69a6317a0 task status = finished >2012-12-17 09:38:17,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended, spm status: SPM >2012-12-17 09:38:17,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e5c64db9-d242-4cde-8c96-d1d69a6317a0), log id: 1dc46dfe >2012-12-17 09:38:17,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMClearTaskVDSCommand, log id: 1dc46dfe >2012-12-17 09:38:17,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b114df6, log id: 72212314 >2012-12-17 09:38:17,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:38:17,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:17,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d6bfdfb >2012-12-17 09:38:17,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d6bfdfb >2012-12-17 09:38:17,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5957bc7a >2012-12-17 09:38:17,817 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:38:17,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3432facc >2012-12-17 09:38:17,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:17,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d9b99d8 >2012-12-17 09:38:17,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d9b99d8 >2012-12-17 09:38:17,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3432facc >2012-12-17 09:38:17,841 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [786aa7bb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:38:26,743 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [f37b622] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:38:26,873 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [130592b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:38:26,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79e9d8 >2012-12-17 09:38:26,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b8b526e >2012-12-17 09:38:26,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:27,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, SpmStopVDSCommand, log id: 4b8b526e >2012-12-17 09:38:27,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, ResetIrsVDSCommand, log id: 79e9d8 >2012-12-17 09:38:27,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [130592b9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77cd7e0c >2012-12-17 09:38:27,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [130592b9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:27,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [130592b9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77cd7e0c >2012-12-17 09:38:27,300 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:38:37,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:38:37,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4917 >2012-12-17 09:38:37,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4917, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bbe182d >2012-12-17 09:38:37,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling started: taskId = b1820aba-b09c-491d-8f7b-52b30b119817 >2012-12-17 09:38:38,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling ended: taskId = b1820aba-b09c-491d-8f7b-52b30b119817 task status = finished >2012-12-17 09:38:38,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] spmStart polling ended, spm status: SPM >2012-12-17 09:38:38,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b1820aba-b09c-491d-8f7b-52b30b119817), log id: 2b1e2ff >2012-12-17 09:38:38,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, HSMClearTaskVDSCommand, log id: 2b1e2ff >2012-12-17 09:38:38,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [40c8f45e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6cadd4ba, log id: bbe182d >2012-12-17 09:38:38,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:38:38,386 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:38:38,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47e51e87 >2012-12-17 09:38:38,444 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [40c8f45e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:38:38,837 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [41718aec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:38:39,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c6d9b2b >2012-12-17 09:38:39,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e047b59 >2012-12-17 09:38:39,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:39,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStopVDSCommand, log id: 7e047b59 >2012-12-17 09:38:39,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, ResetIrsVDSCommand, log id: 7c6d9b2b >2012-12-17 09:38:39,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b67c433 >2012-12-17 09:38:39,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:39,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b67c433 >2012-12-17 09:38:39,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:38:39,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4918 >2012-12-17 09:38:39,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4918, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: eeeec7f >2012-12-17 09:38:39,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling started: taskId = de9cfe28-2953-430a-a97e-21497e909a9e >2012-12-17 09:38:40,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling ended: taskId = de9cfe28-2953-430a-a97e-21497e909a9e task status = finished >2012-12-17 09:38:40,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] spmStart polling ended, spm status: SPM >2012-12-17 09:38:40,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=de9cfe28-2953-430a-a97e-21497e909a9e), log id: 35401ef >2012-12-17 09:38:40,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMClearTaskVDSCommand, log id: 35401ef >2012-12-17 09:38:40,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4af39e, log id: eeeec7f >2012-12-17 09:38:40,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:38:40,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:40,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1e291d7f >2012-12-17 09:38:40,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1e291d7f >2012-12-17 09:38:40,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 47e51e87 >2012-12-17 09:38:40,855 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:38:40,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d4f8c51 >2012-12-17 09:38:40,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:40,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ff391cd >2012-12-17 09:38:40,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ff391cd >2012-12-17 09:38:40,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d4f8c51 >2012-12-17 09:38:40,875 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:38:47,843 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:38:49,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6728cccd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:38:49,597 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [65ba9edd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:38:49,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 21b5abde >2012-12-17 09:38:49,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60dedd2a >2012-12-17 09:38:49,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:49,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, SpmStopVDSCommand, log id: 60dedd2a >2012-12-17 09:38:49,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, ResetIrsVDSCommand, log id: 21b5abde >2012-12-17 09:38:49,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a7f833c >2012-12-17 09:38:49,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:38:49,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [65ba9edd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a7f833c >2012-12-17 09:38:58,063 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:38:59,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:38:59,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4919 >2012-12-17 09:38:59,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4919, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 474b2943 >2012-12-17 09:39:00,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling started: taskId = 923a75bc-7ba6-4a9a-85ec-80e45bc51eda >2012-12-17 09:39:01,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling ended: taskId = 923a75bc-7ba6-4a9a-85ec-80e45bc51eda task status = finished >2012-12-17 09:39:01,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] spmStart polling ended, spm status: SPM >2012-12-17 09:39:01,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=923a75bc-7ba6-4a9a-85ec-80e45bc51eda), log id: 7441893f >2012-12-17 09:39:01,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, HSMClearTaskVDSCommand, log id: 7441893f >2012-12-17 09:39:01,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [134cc8af] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a99c2b9, log id: 474b2943 >2012-12-17 09:39:01,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:01,382 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:39:01,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3ee97ebf >2012-12-17 09:39:01,400 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [134cc8af] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:39:01,506 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [16841b99] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:39:01,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3853af67 >2012-12-17 09:39:01,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 317e1109 >2012-12-17 09:39:01,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:02,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SpmStopVDSCommand, log id: 317e1109 >2012-12-17 09:39:02,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, ResetIrsVDSCommand, log id: 3853af67 >2012-12-17 09:39:02,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13247aac >2012-12-17 09:39:02,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:02,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13247aac >2012-12-17 09:39:02,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:39:02,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4920 >2012-12-17 09:39:02,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4920, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b032fe6 >2012-12-17 09:39:02,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling started: taskId = 9faa6ca9-97d9-449e-839f-1965bcb4cbca >2012-12-17 09:39:03,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended: taskId = 9faa6ca9-97d9-449e-839f-1965bcb4cbca task status = finished >2012-12-17 09:39:03,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] spmStart polling ended, spm status: SPM >2012-12-17 09:39:03,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9faa6ca9-97d9-449e-839f-1965bcb4cbca), log id: 3e3888c6 >2012-12-17 09:39:03,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMClearTaskVDSCommand, log id: 3e3888c6 >2012-12-17 09:39:03,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@719b9f13, log id: 2b032fe6 >2012-12-17 09:39:03,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:03,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:03,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 44b929ee >2012-12-17 09:39:03,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44b929ee >2012-12-17 09:39:03,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3ee97ebf >2012-12-17 09:39:03,521 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:39:03,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 586feb81 >2012-12-17 09:39:03,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:03,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a5f0eb4 >2012-12-17 09:39:03,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a5f0eb4 >2012-12-17 09:39:03,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [7790038] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 586feb81 >2012-12-17 09:39:03,541 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [7790038] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:39:08,299 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:39:12,421 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [6157bb5e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:39:12,602 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [4e8c7167] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:39:12,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 39b12985 >2012-12-17 09:39:12,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 18389e7 >2012-12-17 09:39:12,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:12,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, SpmStopVDSCommand, log id: 18389e7 >2012-12-17 09:39:12,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, ResetIrsVDSCommand, log id: 39b12985 >2012-12-17 09:39:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a340a83 >2012-12-17 09:39:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:12,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [4e8c7167] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a340a83 >2012-12-17 09:39:18,488 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:39:23,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:39:23,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4921 >2012-12-17 09:39:23,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4921, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f958afd >2012-12-17 09:39:23,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling started: taskId = 94eed677-a039-46da-9c4a-d838648dffde >2012-12-17 09:39:24,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling ended: taskId = 94eed677-a039-46da-9c4a-d838648dffde task status = finished >2012-12-17 09:39:24,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] spmStart polling ended, spm status: SPM >2012-12-17 09:39:24,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=94eed677-a039-46da-9c4a-d838648dffde), log id: 41474783 >2012-12-17 09:39:24,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, HSMClearTaskVDSCommand, log id: 41474783 >2012-12-17 09:39:24,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [6c33f0df] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ea80af0, log id: 1f958afd >2012-12-17 09:39:24,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:24,154 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:39:24,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2380c43f >2012-12-17 09:39:24,187 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [6c33f0df] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:39:24,318 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [56fd87e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:39:24,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ced27b4 >2012-12-17 09:39:24,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3fa72742 >2012-12-17 09:39:24,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:24,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStopVDSCommand, log id: 3fa72742 >2012-12-17 09:39:24,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, ResetIrsVDSCommand, log id: 5ced27b4 >2012-12-17 09:39:24,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15e2412 >2012-12-17 09:39:24,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:24,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15e2412 >2012-12-17 09:39:24,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:39:24,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4922 >2012-12-17 09:39:24,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4922, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5023ea79 >2012-12-17 09:39:24,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling started: taskId = 00247663-9254-464d-85c5-794d10c6dd57 >2012-12-17 09:39:25,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended: taskId = 00247663-9254-464d-85c5-794d10c6dd57 task status = finished >2012-12-17 09:39:25,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended, spm status: SPM >2012-12-17 09:39:25,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=00247663-9254-464d-85c5-794d10c6dd57), log id: 17205f18 >2012-12-17 09:39:25,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMClearTaskVDSCommand, log id: 17205f18 >2012-12-17 09:39:25,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e432c84, log id: 5023ea79 >2012-12-17 09:39:26,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:26,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:26,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5403a555 >2012-12-17 09:39:26,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5403a555 >2012-12-17 09:39:26,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2380c43f >2012-12-17 09:39:26,468 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:39:26,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b5b9a8a >2012-12-17 09:39:26,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:26,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2565775e >2012-12-17 09:39:26,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2565775e >2012-12-17 09:39:26,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b5b9a8a >2012-12-17 09:39:26,489 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:39:28,770 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:39:34,751 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [8d9c42c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:39:34,911 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [34a366b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:39:35,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 228c02e9 >2012-12-17 09:39:35,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ce32a47 >2012-12-17 09:39:35,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:35,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStopVDSCommand, log id: 4ce32a47 >2012-12-17 09:39:35,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, ResetIrsVDSCommand, log id: 228c02e9 >2012-12-17 09:39:35,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5dbf1440 >2012-12-17 09:39:35,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:35,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5dbf1440 >2012-12-17 09:39:39,018 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:39:45,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:39:45,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4923 >2012-12-17 09:39:45,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4923, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5926b6d9 >2012-12-17 09:39:45,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling started: taskId = 6f469cfe-def8-45d4-a6da-f2f7e6a49b63 >2012-12-17 09:39:46,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended: taskId = 6f469cfe-def8-45d4-a6da-f2f7e6a49b63 task status = finished >2012-12-17 09:39:46,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] spmStart polling ended, spm status: SPM >2012-12-17 09:39:46,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f469cfe-def8-45d4-a6da-f2f7e6a49b63), log id: 6d0d3e4c >2012-12-17 09:39:46,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, HSMClearTaskVDSCommand, log id: 6d0d3e4c >2012-12-17 09:39:46,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [1240becc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@752783, log id: 5926b6d9 >2012-12-17 09:39:47,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:47,864 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:39:47,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 50491293 >2012-12-17 09:39:47,977 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1240becc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:39:49,727 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:39:49,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 226b3738 >2012-12-17 09:39:49,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23a21faf >2012-12-17 09:39:49,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:49,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, SpmStopVDSCommand, log id: 23a21faf >2012-12-17 09:39:50,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, ResetIrsVDSCommand, log id: 226b3738 >2012-12-17 09:39:50,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 329ef07e >2012-12-17 09:39:50,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:50,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 329ef07e >2012-12-17 09:39:50,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:39:50,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4924 >2012-12-17 09:39:50,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4924, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 61d86699 >2012-12-17 09:39:50,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling started: taskId = b219dc71-6476-445d-a052-1fa3348e6cb8 >2012-12-17 09:39:51,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended: taskId = b219dc71-6476-445d-a052-1fa3348e6cb8 task status = finished >2012-12-17 09:39:51,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] spmStart polling ended, spm status: SPM >2012-12-17 09:39:51,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b219dc71-6476-445d-a052-1fa3348e6cb8), log id: 2802bea5 >2012-12-17 09:39:51,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMClearTaskVDSCommand, log id: 2802bea5 >2012-12-17 09:39:51,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c325728, log id: 61d86699 >2012-12-17 09:39:51,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:39:52,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:52,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2356f486 >2012-12-17 09:39:52,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2356f486 >2012-12-17 09:39:52,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50491293 >2012-12-17 09:39:52,209 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:39:52,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21cf9b8a >2012-12-17 09:39:52,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:39:52,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7948415b >2012-12-17 09:39:52,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7948415b >2012-12-17 09:39:52,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-100) [6bb29080] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21cf9b8a >2012-12-17 09:39:52,229 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) [6bb29080] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Checking autorecoverable hosts >2012-12-17 09:40:00,094 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Autorecovering 0 hosts >2012-12-17 09:40:00,094 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Checking autorecoverable hosts done >2012-12-17 09:40:00,094 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [24a780e1] Autorecovering storage domains is disabled, skipping >2012-12-17 09:40:00,279 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:00,359 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [1ed39251] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:40:02,670 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:40:02,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 210c0d3e >2012-12-17 09:40:02,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 33618c62 >2012-12-17 09:40:02,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:03,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, SpmStopVDSCommand, log id: 33618c62 >2012-12-17 09:40:03,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, ResetIrsVDSCommand, log id: 210c0d3e >2012-12-17 09:40:03,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4de7108b >2012-12-17 09:40:03,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:03,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4de7108b >2012-12-17 09:40:11,389 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:13,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:40:13,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4925 >2012-12-17 09:40:13,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4925, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a31f6af >2012-12-17 09:40:13,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling started: taskId = 694138e5-811e-499a-a432-6e619c50baee >2012-12-17 09:40:14,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling ended: taskId = 694138e5-811e-499a-a432-6e619c50baee task status = finished >2012-12-17 09:40:14,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] spmStart polling ended, spm status: SPM >2012-12-17 09:40:14,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=694138e5-811e-499a-a432-6e619c50baee), log id: 5de11455 >2012-12-17 09:40:14,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, HSMClearTaskVDSCommand, log id: 5de11455 >2012-12-17 09:40:14,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [56a0d824] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63fe0899, log id: 2a31f6af >2012-12-17 09:40:14,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:40:14,466 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:40:14,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40ff0f06 >2012-12-17 09:40:14,486 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [56a0d824] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:40:14,602 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [7c8941c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:40:14,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c5f9f23 >2012-12-17 09:40:14,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4245b538 >2012-12-17 09:40:14,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:14,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SpmStopVDSCommand, log id: 4245b538 >2012-12-17 09:40:14,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, ResetIrsVDSCommand, log id: 7c5f9f23 >2012-12-17 09:40:14,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fe70786 >2012-12-17 09:40:14,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:14,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fe70786 >2012-12-17 09:40:14,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:40:14,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4926 >2012-12-17 09:40:14,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4926, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 513a74f5 >2012-12-17 09:40:14,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling started: taskId = 9be41fa0-52a2-43e5-8498-9d0b202bdf70 >2012-12-17 09:40:15,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended: taskId = 9be41fa0-52a2-43e5-8498-9d0b202bdf70 task status = finished >2012-12-17 09:40:15,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] spmStart polling ended, spm status: SPM >2012-12-17 09:40:15,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9be41fa0-52a2-43e5-8498-9d0b202bdf70), log id: 175563b1 >2012-12-17 09:40:16,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMClearTaskVDSCommand, log id: 175563b1 >2012-12-17 09:40:16,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6704056e, log id: 513a74f5 >2012-12-17 09:40:16,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:40:16,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:16,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16ae9298 >2012-12-17 09:40:16,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16ae9298 >2012-12-17 09:40:16,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40ff0f06 >2012-12-17 09:40:16,724 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:40:16,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5365997b >2012-12-17 09:40:16,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:16,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b3756f9 >2012-12-17 09:40:16,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b3756f9 >2012-12-17 09:40:16,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-66) [492c9849] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5365997b >2012-12-17 09:40:16,745 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-66) [492c9849] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:40:21,830 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:24,962 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [3a1c2706] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:40:25,087 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [4549d797] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:40:25,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40398b69 >2012-12-17 09:40:25,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7c325aff >2012-12-17 09:40:25,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:25,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, SpmStopVDSCommand, log id: 7c325aff >2012-12-17 09:40:25,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, ResetIrsVDSCommand, log id: 40398b69 >2012-12-17 09:40:25,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [4549d797] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30072f7e >2012-12-17 09:40:25,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4549d797] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:25,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [4549d797] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30072f7e >2012-12-17 09:40:32,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:36,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:40:36,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4927 >2012-12-17 09:40:36,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4927, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 135a16aa >2012-12-17 09:40:36,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling started: taskId = 013c5443-5ac7-413d-803d-ed2984f7a0a0 >2012-12-17 09:40:37,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling ended: taskId = 013c5443-5ac7-413d-803d-ed2984f7a0a0 task status = finished >2012-12-17 09:40:37,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] spmStart polling ended, spm status: SPM >2012-12-17 09:40:37,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=013c5443-5ac7-413d-803d-ed2984f7a0a0), log id: 467c073a >2012-12-17 09:40:37,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, HSMClearTaskVDSCommand, log id: 467c073a >2012-12-17 09:40:37,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [280f54d2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78be94fc, log id: 135a16aa >2012-12-17 09:40:37,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:40:37,864 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:40:37,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31a53e04 >2012-12-17 09:40:37,870 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [280f54d2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:40:37,979 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [6999432f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:40:38,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: cbea4cf >2012-12-17 09:40:38,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50ed6a7d >2012-12-17 09:40:38,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:38,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SpmStopVDSCommand, log id: 50ed6a7d >2012-12-17 09:40:38,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, ResetIrsVDSCommand, log id: cbea4cf >2012-12-17 09:40:38,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43e7decd >2012-12-17 09:40:38,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:38,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43e7decd >2012-12-17 09:40:38,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:40:38,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4928 >2012-12-17 09:40:38,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4928, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45c62a06 >2012-12-17 09:40:38,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = a71467b6-5896-435a-abee-69dcd4449a38 >2012-12-17 09:40:39,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = a71467b6-5896-435a-abee-69dcd4449a38 task status = finished >2012-12-17 09:40:39,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM >2012-12-17 09:40:39,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a71467b6-5896-435a-abee-69dcd4449a38), log id: 5f8816d7 >2012-12-17 09:40:39,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 5f8816d7 >2012-12-17 09:40:39,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f93ae40, log id: 45c62a06 >2012-12-17 09:40:39,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:40:39,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:39,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 61eb85ae >2012-12-17 09:40:39,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 61eb85ae >2012-12-17 09:40:39,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31a53e04 >2012-12-17 09:40:39,478 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:40:39,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3757985e >2012-12-17 09:40:39,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:39,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a813577 >2012-12-17 09:40:39,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a813577 >2012-12-17 09:40:39,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3757985e >2012-12-17 09:40:39,497 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:40:42,411 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:48,375 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [24a780e1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:40:48,558 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [511011e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:40:48,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 42b3a8dd >2012-12-17 09:40:48,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49f0fe06 >2012-12-17 09:40:48,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:49,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, SpmStopVDSCommand, log id: 49f0fe06 >2012-12-17 09:40:49,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, ResetIrsVDSCommand, log id: 42b3a8dd >2012-12-17 09:40:49,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 422b1202 >2012-12-17 09:40:49,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:40:49,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 422b1202 >2012-12-17 09:40:52,646 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:40:59,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:40:59,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4929 >2012-12-17 09:40:59,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4929, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2644b0d2 >2012-12-17 09:40:59,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling started: taskId = cdd22f2c-daaf-4980-8655-4cf7a396558a >2012-12-17 09:41:00,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling ended: taskId = cdd22f2c-daaf-4980-8655-4cf7a396558a task status = finished >2012-12-17 09:41:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] spmStart polling ended, spm status: SPM >2012-12-17 09:41:00,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cdd22f2c-daaf-4980-8655-4cf7a396558a), log id: 5048f852 >2012-12-17 09:41:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, HSMClearTaskVDSCommand, log id: 5048f852 >2012-12-17 09:41:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [76c72cb4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66a6f8ab, log id: 2644b0d2 >2012-12-17 09:41:01,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:02,193 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:41:02,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42e94f89 >2012-12-17 09:41:02,307 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [76c72cb4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:41:02,579 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:41:02,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f4915e >2012-12-17 09:41:02,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f94a078 >2012-12-17 09:41:02,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:02,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SpmStopVDSCommand, log id: 1f94a078 >2012-12-17 09:41:02,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, ResetIrsVDSCommand, log id: 48f4915e >2012-12-17 09:41:02,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e9a3db7 >2012-12-17 09:41:02,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:02,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e9a3db7 >2012-12-17 09:41:02,956 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:41:03,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:41:03,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4930 >2012-12-17 09:41:03,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4930, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b4a9679 >2012-12-17 09:41:03,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling started: taskId = f7293709-41ed-403b-8c89-f2bde3dcc0d3 >2012-12-17 09:41:04,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended: taskId = f7293709-41ed-403b-8c89-f2bde3dcc0d3 task status = finished >2012-12-17 09:41:04,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended, spm status: SPM >2012-12-17 09:41:04,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7293709-41ed-403b-8c89-f2bde3dcc0d3), log id: 6cfb447d >2012-12-17 09:41:04,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMClearTaskVDSCommand, log id: 6cfb447d >2012-12-17 09:41:04,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@644da270, log id: 6b4a9679 >2012-12-17 09:41:04,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:04,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:04,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65130b82 >2012-12-17 09:41:04,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65130b82 >2012-12-17 09:41:04,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42e94f89 >2012-12-17 09:41:04,192 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:04,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4558b7a5 >2012-12-17 09:41:04,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:04,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22bf19c4 >2012-12-17 09:41:04,194 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:41:04,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5027b43b >2012-12-17 09:41:04,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22bf19c4 >2012-12-17 09:41:04,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4558b7a5 >2012-12-17 09:41:04,212 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [2c554bbe] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:04,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:04,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d6e0662 >2012-12-17 09:41:04,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d6e0662 >2012-12-17 09:41:04,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5027b43b >2012-12-17 09:41:04,232 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:04,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1265233b >2012-12-17 09:41:04,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:04,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6cdffb >2012-12-17 09:41:04,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6cdffb >2012-12-17 09:41:04,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-81) [5117cf2d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1265233b >2012-12-17 09:41:04,252 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-81) [5117cf2d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:13,033 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [6c899ac8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:41:13,143 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:41:13,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6871356b >2012-12-17 09:41:13,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b4f351a >2012-12-17 09:41:13,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:13,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, SpmStopVDSCommand, log id: b4f351a >2012-12-17 09:41:13,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, ResetIrsVDSCommand, log id: 6871356b >2012-12-17 09:41:13,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e5a90e1 >2012-12-17 09:41:13,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:13,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e5a90e1 >2012-12-17 09:41:23,448 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:41:23,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:41:23,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4931 >2012-12-17 09:41:23,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4931, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53dc932 >2012-12-17 09:41:23,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling started: taskId = e8b15d7f-e384-4cd9-90e4-aaee968bba41 >2012-12-17 09:41:24,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling ended: taskId = e8b15d7f-e384-4cd9-90e4-aaee968bba41 task status = finished >2012-12-17 09:41:24,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] spmStart polling ended, spm status: SPM >2012-12-17 09:41:24,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e8b15d7f-e384-4cd9-90e4-aaee968bba41), log id: 60d54f2c >2012-12-17 09:41:24,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, HSMClearTaskVDSCommand, log id: 60d54f2c >2012-12-17 09:41:24,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5441035e, log id: 53dc932 >2012-12-17 09:41:24,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:24,800 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:41:24,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 593bd992 >2012-12-17 09:41:24,821 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2b6f5c8b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:41:24,947 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [21d8114] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:41:25,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12f7da04 >2012-12-17 09:41:25,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7188f6da >2012-12-17 09:41:25,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:25,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, SpmStopVDSCommand, log id: 7188f6da >2012-12-17 09:41:25,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, ResetIrsVDSCommand, log id: 12f7da04 >2012-12-17 09:41:25,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9098fb >2012-12-17 09:41:25,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:25,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9098fb >2012-12-17 09:41:25,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:41:25,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4932 >2012-12-17 09:41:25,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4932, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 457f935b >2012-12-17 09:41:25,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling started: taskId = 8cd371fd-6acd-4ce3-bce9-916bdf7e7da3 >2012-12-17 09:41:26,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling ended: taskId = 8cd371fd-6acd-4ce3-bce9-916bdf7e7da3 task status = finished >2012-12-17 09:41:26,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] spmStart polling ended, spm status: SPM >2012-12-17 09:41:26,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8cd371fd-6acd-4ce3-bce9-916bdf7e7da3), log id: 2cf50cfe >2012-12-17 09:41:26,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMClearTaskVDSCommand, log id: 2cf50cfe >2012-12-17 09:41:26,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c280ec1, log id: 457f935b >2012-12-17 09:41:26,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:26,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:26,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 186ae989 >2012-12-17 09:41:26,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 186ae989 >2012-12-17 09:41:26,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 593bd992 >2012-12-17 09:41:26,500 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:26,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 130a2de5 >2012-12-17 09:41:26,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:26,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2e921bf7 >2012-12-17 09:41:26,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2e921bf7 >2012-12-17 09:41:26,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-74) [16841b99] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 130a2de5 >2012-12-17 09:41:26,520 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) [16841b99] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:33,633 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:41:35,409 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7db92791] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:41:36,169 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [21122b75] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:41:36,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 107b686e >2012-12-17 09:41:36,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 758722bc >2012-12-17 09:41:36,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:37,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStopVDSCommand, log id: 758722bc >2012-12-17 09:41:37,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, ResetIrsVDSCommand, log id: 107b686e >2012-12-17 09:41:37,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6215b8c8 >2012-12-17 09:41:37,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:37,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6215b8c8 >2012-12-17 09:41:43,849 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:41:47,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:41:47,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4933 >2012-12-17 09:41:47,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4933, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ed4044 >2012-12-17 09:41:47,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling started: taskId = 968a8e2c-8e23-4ce1-b7c7-4014d9592311 >2012-12-17 09:41:49,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling ended: taskId = 968a8e2c-8e23-4ce1-b7c7-4014d9592311 task status = finished >2012-12-17 09:41:49,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] spmStart polling ended, spm status: SPM >2012-12-17 09:41:49,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=968a8e2c-8e23-4ce1-b7c7-4014d9592311), log id: 35d804fb >2012-12-17 09:41:49,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, HSMClearTaskVDSCommand, log id: 35d804fb >2012-12-17 09:41:49,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [588a5ec3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@792f165a, log id: 6ed4044 >2012-12-17 09:41:49,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:49,565 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:41:49,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c3e7702 >2012-12-17 09:41:49,598 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [588a5ec3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:41:49,762 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [2420ee59] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:41:49,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 31e2f4ec >2012-12-17 09:41:49,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5654a1db >2012-12-17 09:41:49,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:49,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, SpmStopVDSCommand, log id: 5654a1db >2012-12-17 09:41:49,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, ResetIrsVDSCommand, log id: 31e2f4ec >2012-12-17 09:41:50,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66d687a7 >2012-12-17 09:41:50,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:50,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2420ee59] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66d687a7 >2012-12-17 09:41:50,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:41:50,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4934 >2012-12-17 09:41:50,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4934, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23733f42 >2012-12-17 09:41:50,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling started: taskId = def68268-145d-4162-ac04-a39494ddcf1f >2012-12-17 09:41:51,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended: taskId = def68268-145d-4162-ac04-a39494ddcf1f task status = finished >2012-12-17 09:41:51,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended, spm status: SPM >2012-12-17 09:41:51,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=def68268-145d-4162-ac04-a39494ddcf1f), log id: 69e87359 >2012-12-17 09:41:51,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMClearTaskVDSCommand, log id: 69e87359 >2012-12-17 09:41:51,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bed0e61, log id: 23733f42 >2012-12-17 09:41:51,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:41:51,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:51,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3aa1bb7e >2012-12-17 09:41:51,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3aa1bb7e >2012-12-17 09:41:51,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c3e7702 >2012-12-17 09:41:51,629 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:51,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 501c6b44 >2012-12-17 09:41:51,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:41:51,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4eae36d9 >2012-12-17 09:41:51,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4eae36d9 >2012-12-17 09:41:51,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 501c6b44 >2012-12-17 09:41:51,654 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [41718aec] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:41:54,076 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:42:00,138 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3623f38c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:00,349 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [44cd9139] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:00,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 18f8c8a0 >2012-12-17 09:42:00,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 166cbf3b >2012-12-17 09:42:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:00,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStopVDSCommand, log id: 166cbf3b >2012-12-17 09:42:01,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, ResetIrsVDSCommand, log id: 18f8c8a0 >2012-12-17 09:42:01,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 561a060c >2012-12-17 09:42:01,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:01,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 561a060c >2012-12-17 09:42:04,315 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:42:11,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:11,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4935 >2012-12-17 09:42:11,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4935, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e7a3a42 >2012-12-17 09:42:11,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling started: taskId = 9a6bcbbe-2719-4e69-943e-8cc942e23731 >2012-12-17 09:42:12,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended: taskId = 9a6bcbbe-2719-4e69-943e-8cc942e23731 task status = finished >2012-12-17 09:42:12,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] spmStart polling ended, spm status: SPM >2012-12-17 09:42:12,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9a6bcbbe-2719-4e69-943e-8cc942e23731), log id: 7428c28d >2012-12-17 09:42:12,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, HSMClearTaskVDSCommand, log id: 7428c28d >2012-12-17 09:42:12,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [3ff8505] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ffcf28f, log id: 3e7a3a42 >2012-12-17 09:42:12,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:42:12,970 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:42:12,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4723ad5e >2012-12-17 09:42:13,015 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [3ff8505] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:13,383 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [58bd42f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:13,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 47b65888 >2012-12-17 09:42:13,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50bcb4bc >2012-12-17 09:42:13,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:13,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, SpmStopVDSCommand, log id: 50bcb4bc >2012-12-17 09:42:13,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, ResetIrsVDSCommand, log id: 47b65888 >2012-12-17 09:42:13,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40d609cd >2012-12-17 09:42:13,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:13,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40d609cd >2012-12-17 09:42:13,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:13,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4936 >2012-12-17 09:42:13,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4936, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 19fb0bdd >2012-12-17 09:42:13,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling started: taskId = 6978f336-1b1b-42f4-b0e0-13b95f38c1f8 >2012-12-17 09:42:14,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling ended: taskId = 6978f336-1b1b-42f4-b0e0-13b95f38c1f8 task status = finished >2012-12-17 09:42:14,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] spmStart polling ended, spm status: SPM >2012-12-17 09:42:14,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6978f336-1b1b-42f4-b0e0-13b95f38c1f8), log id: 73ed37b7 >2012-12-17 09:42:14,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMClearTaskVDSCommand, log id: 73ed37b7 >2012-12-17 09:42:14,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ae50ad6, log id: 19fb0bdd >2012-12-17 09:42:14,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:42:14,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:14,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3a249275 >2012-12-17 09:42:15,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3a249275 >2012-12-17 09:42:15,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4723ad5e >2012-12-17 09:42:15,005 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:42:15,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 370b5d25 >2012-12-17 09:42:15,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:15,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 751db5e5 >2012-12-17 09:42:15,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 751db5e5 >2012-12-17 09:42:15,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [6999432f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 370b5d25 >2012-12-17 09:42:15,025 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [6999432f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:42:23,886 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [136706cf] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:24,062 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [17eab1eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:24,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d8d0dfa >2012-12-17 09:42:24,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b736393 >2012-12-17 09:42:24,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:24,711 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:42:24,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SpmStopVDSCommand, log id: 7b736393 >2012-12-17 09:42:24,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, ResetIrsVDSCommand, log id: 7d8d0dfa >2012-12-17 09:42:24,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3afbe695 >2012-12-17 09:42:24,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:24,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3afbe695 >2012-12-17 09:42:34,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:34,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4937 >2012-12-17 09:42:34,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4937, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8d149eb >2012-12-17 09:42:34,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling started: taskId = 01403bdd-352d-433d-a458-0ee2bd5d5814 >2012-12-17 09:42:35,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling ended: taskId = 01403bdd-352d-433d-a458-0ee2bd5d5814 task status = finished >2012-12-17 09:42:35,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] spmStart polling ended, spm status: SPM >2012-12-17 09:42:35,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=01403bdd-352d-433d-a458-0ee2bd5d5814), log id: 203d1c41 >2012-12-17 09:42:35,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, HSMClearTaskVDSCommand, log id: 203d1c41 >2012-12-17 09:42:35,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [46833f1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c09fff0, log id: 8d149eb >2012-12-17 09:42:36,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:42:36,368 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:42:36,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f018369 >2012-12-17 09:42:36,385 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [46833f1f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:36,516 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:36,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36bbeb2b >2012-12-17 09:42:36,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c106166 >2012-12-17 09:42:36,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:37,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStopVDSCommand, log id: c106166 >2012-12-17 09:42:37,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, ResetIrsVDSCommand, log id: 36bbeb2b >2012-12-17 09:42:37,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d34ec4 >2012-12-17 09:42:37,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:37,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d34ec4 >2012-12-17 09:42:37,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:37,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4938 >2012-12-17 09:42:37,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4938, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 546cf249 >2012-12-17 09:42:37,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = 68db3bbe-2f67-4ac5-baca-aef8c39da9bb >2012-12-17 09:42:38,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = 68db3bbe-2f67-4ac5-baca-aef8c39da9bb task status = finished >2012-12-17 09:42:38,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM >2012-12-17 09:42:38,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=68db3bbe-2f67-4ac5-baca-aef8c39da9bb), log id: 6c3a00d9 >2012-12-17 09:42:38,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 6c3a00d9 >2012-12-17 09:42:38,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@540dde11, log id: 546cf249 >2012-12-17 09:42:38,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:42:38,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:38,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7de43fbb >2012-12-17 09:42:38,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7de43fbb >2012-12-17 09:42:38,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f018369 >2012-12-17 09:42:38,579 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:42:38,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c99f627 >2012-12-17 09:42:38,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:38,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: a14711d >2012-12-17 09:42:38,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: a14711d >2012-12-17 09:42:38,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c99f627 >2012-12-17 09:42:38,599 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:42:45,142 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:42:47,366 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5c6bc06e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:47,469 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [6412070c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:47,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 23707a2a >2012-12-17 09:42:47,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 74a5b561 >2012-12-17 09:42:47,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:47,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, SpmStopVDSCommand, log id: 74a5b561 >2012-12-17 09:42:47,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, ResetIrsVDSCommand, log id: 23707a2a >2012-12-17 09:42:47,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34dc3f45 >2012-12-17 09:42:47,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:47,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34dc3f45 >2012-12-17 09:42:56,789 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:42:57,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:57,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4939 >2012-12-17 09:42:57,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4939, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5530bba8 >2012-12-17 09:42:57,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling started: taskId = 48b429bd-8996-4b0d-adbf-9699f7a1b682 >2012-12-17 09:42:58,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling ended: taskId = 48b429bd-8996-4b0d-adbf-9699f7a1b682 task status = finished >2012-12-17 09:42:58,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] spmStart polling ended, spm status: SPM >2012-12-17 09:42:58,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=48b429bd-8996-4b0d-adbf-9699f7a1b682), log id: 2079d445 >2012-12-17 09:42:58,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, HSMClearTaskVDSCommand, log id: 2079d445 >2012-12-17 09:42:58,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [9b1d86] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e5f0eab, log id: 5530bba8 >2012-12-17 09:42:58,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:42:58,995 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:42:58,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6518bb61 >2012-12-17 09:42:59,011 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [9b1d86] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:42:59,269 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [24cb4742] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:42:59,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75373532 >2012-12-17 09:42:59,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67447da2 >2012-12-17 09:42:59,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:59,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SpmStopVDSCommand, log id: 67447da2 >2012-12-17 09:42:59,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, ResetIrsVDSCommand, log id: 75373532 >2012-12-17 09:42:59,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 761ab201 >2012-12-17 09:42:59,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:42:59,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 761ab201 >2012-12-17 09:42:59,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:42:59,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4940 >2012-12-17 09:42:59,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4940, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d76600c >2012-12-17 09:42:59,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 6e89644f-6aed-45f4-a4e4-5e453b9b0b3a >2012-12-17 09:43:00,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 6e89644f-6aed-45f4-a4e4-5e453b9b0b3a task status = finished >2012-12-17 09:43:00,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM >2012-12-17 09:43:00,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6e89644f-6aed-45f4-a4e4-5e453b9b0b3a), log id: 7a05e4a0 >2012-12-17 09:43:00,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 7a05e4a0 >2012-12-17 09:43:00,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18e502f6, log id: 6d76600c >2012-12-17 09:43:03,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:43:03,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:03,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22303e79 >2012-12-17 09:43:03,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22303e79 >2012-12-17 09:43:03,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6518bb61 >2012-12-17 09:43:03,901 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:03,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d558b78 >2012-12-17 09:43:03,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:03,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2b54930f >2012-12-17 09:43:03,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b54930f >2012-12-17 09:43:03,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d558b78 >2012-12-17 09:43:03,921 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [1ab078c7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:07,016 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:43:09,717 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2bb64a93] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:43:09,808 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [c06c0c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:43:09,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 222ee3b3 >2012-12-17 09:43:09,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 14366847 >2012-12-17 09:43:09,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:10,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStopVDSCommand, log id: 14366847 >2012-12-17 09:43:10,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, ResetIrsVDSCommand, log id: 222ee3b3 >2012-12-17 09:43:10,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f4b5afa >2012-12-17 09:43:10,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:10,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f4b5afa >2012-12-17 09:43:17,210 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:43:20,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:43:20,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4941 >2012-12-17 09:43:20,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4941, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62a19d2c >2012-12-17 09:43:20,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling started: taskId = ccfdcad7-a4dc-4753-801c-d320e9527ee1 >2012-12-17 09:43:21,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling ended: taskId = ccfdcad7-a4dc-4753-801c-d320e9527ee1 task status = finished >2012-12-17 09:43:21,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] spmStart polling ended, spm status: SPM >2012-12-17 09:43:21,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ccfdcad7-a4dc-4753-801c-d320e9527ee1), log id: 647c3569 >2012-12-17 09:43:21,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, HSMClearTaskVDSCommand, log id: 647c3569 >2012-12-17 09:43:21,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [5d1b6c83] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1696b409, log id: 62a19d2c >2012-12-17 09:43:21,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:43:21,297 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:43:21,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3053d947 >2012-12-17 09:43:21,339 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [5d1b6c83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:43:21,447 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [1eb0b032] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:43:21,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d1c6315 >2012-12-17 09:43:21,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24703341 >2012-12-17 09:43:21,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:21,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, SpmStopVDSCommand, log id: 24703341 >2012-12-17 09:43:21,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, ResetIrsVDSCommand, log id: 7d1c6315 >2012-12-17 09:43:21,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46809868 >2012-12-17 09:43:21,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:21,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [1eb0b032] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46809868 >2012-12-17 09:43:22,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:43:22,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4942 >2012-12-17 09:43:22,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4942, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e1e2f3d >2012-12-17 09:43:22,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling started: taskId = 0723a8af-cd03-4793-acc7-1b2eb4001990 >2012-12-17 09:43:23,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended: taskId = 0723a8af-cd03-4793-acc7-1b2eb4001990 task status = finished >2012-12-17 09:43:23,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended, spm status: SPM >2012-12-17 09:43:23,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0723a8af-cd03-4793-acc7-1b2eb4001990), log id: 760850b1 >2012-12-17 09:43:23,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMClearTaskVDSCommand, log id: 760850b1 >2012-12-17 09:43:23,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1be9d101, log id: 3e1e2f3d >2012-12-17 09:43:23,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:43:23,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:23,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55e4e851 >2012-12-17 09:43:23,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e4e851 >2012-12-17 09:43:23,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3053d947 >2012-12-17 09:43:23,931 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:23,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8236d8d >2012-12-17 09:43:23,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:23,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60dd7b1 >2012-12-17 09:43:23,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60dd7b1 >2012-12-17 09:43:23,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8236d8d >2012-12-17 09:43:23,952 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [127bfc] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:34,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [492c9849] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:43:34,538 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:43:34,863 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [205b7df7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:43:35,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49b76ca2 >2012-12-17 09:43:35,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1dbee4ec >2012-12-17 09:43:35,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:35,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, SpmStopVDSCommand, log id: 1dbee4ec >2012-12-17 09:43:35,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, ResetIrsVDSCommand, log id: 49b76ca2 >2012-12-17 09:43:35,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48471be5 >2012-12-17 09:43:35,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:35,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48471be5 >2012-12-17 09:43:45,113 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:43:45,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:43:46,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4943 >2012-12-17 09:43:46,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4943, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22480666 >2012-12-17 09:43:46,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling started: taskId = d3f91098-761e-4218-b7fc-379c60df049a >2012-12-17 09:43:47,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling ended: taskId = d3f91098-761e-4218-b7fc-379c60df049a task status = finished >2012-12-17 09:43:47,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] spmStart polling ended, spm status: SPM >2012-12-17 09:43:47,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [248935dc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d3f91098-761e-4218-b7fc-379c60df049a), log id: 1d26c9e0 >2012-12-17 09:43:47,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, HSMClearTaskVDSCommand, log id: 1d26c9e0 >2012-12-17 09:43:47,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [248935dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d8fa013, log id: 22480666 >2012-12-17 09:43:47,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:43:47,877 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:43:47,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1eb2e2e7 >2012-12-17 09:43:47,891 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [248935dc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:43:48,002 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [4d6b231f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:43:48,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 706f8ae9 >2012-12-17 09:43:48,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 29835bd3 >2012-12-17 09:43:48,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:48,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, SpmStopVDSCommand, log id: 29835bd3 >2012-12-17 09:43:48,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, ResetIrsVDSCommand, log id: 706f8ae9 >2012-12-17 09:43:48,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bcd97cf >2012-12-17 09:43:48,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [4d6b231f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:48,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [4d6b231f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bcd97cf >2012-12-17 09:43:48,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:43:48,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4944 >2012-12-17 09:43:48,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4944, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 475c3b06 >2012-12-17 09:43:48,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling started: taskId = db62bd5f-9100-4fe9-93c5-0aa612fdf22f >2012-12-17 09:43:49,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended: taskId = db62bd5f-9100-4fe9-93c5-0aa612fdf22f task status = finished >2012-12-17 09:43:49,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended, spm status: SPM >2012-12-17 09:43:49,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db62bd5f-9100-4fe9-93c5-0aa612fdf22f), log id: 5b204280 >2012-12-17 09:43:49,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMClearTaskVDSCommand, log id: 5b204280 >2012-12-17 09:43:49,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22ee103d, log id: 475c3b06 >2012-12-17 09:43:49,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:43:49,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:49,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6fb5f1e7 >2012-12-17 09:43:49,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6fb5f1e7 >2012-12-17 09:43:49,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1eb2e2e7 >2012-12-17 09:43:49,663 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:49,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 121cadfd >2012-12-17 09:43:49,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:49,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56769e04 >2012-12-17 09:43:49,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56769e04 >2012-12-17 09:43:49,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 121cadfd >2012-12-17 09:43:49,683 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-60) [6eb2e264] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:43:55,483 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:43:58,461 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [6999432f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:43:58,562 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [75733504] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:43:58,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1df005d8 >2012-12-17 09:43:58,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 121c9c73 >2012-12-17 09:43:58,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:58,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStopVDSCommand, log id: 121c9c73 >2012-12-17 09:43:58,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, ResetIrsVDSCommand, log id: 1df005d8 >2012-12-17 09:43:58,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 691e46ec >2012-12-17 09:43:58,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:43:58,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 691e46ec >2012-12-17 09:44:07,339 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:44:08,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:44:09,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4945 >2012-12-17 09:44:09,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4945, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e581f7a >2012-12-17 09:44:09,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling started: taskId = 157b816a-85b7-4501-9bc5-a3bbe928f473 >2012-12-17 09:44:10,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling ended: taskId = 157b816a-85b7-4501-9bc5-a3bbe928f473 task status = finished >2012-12-17 09:44:10,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] spmStart polling ended, spm status: SPM >2012-12-17 09:44:10,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49192055] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=157b816a-85b7-4501-9bc5-a3bbe928f473), log id: 44d47477 >2012-12-17 09:44:10,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, HSMClearTaskVDSCommand, log id: 44d47477 >2012-12-17 09:44:10,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49192055] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@582a53ee, log id: 5e581f7a >2012-12-17 09:44:10,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:44:10,287 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:44:10,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6146335d >2012-12-17 09:44:10,300 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49192055] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:44:10,542 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [61938c83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:44:13,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 54857515 >2012-12-17 09:44:13,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4af712bd >2012-12-17 09:44:13,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:13,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, SpmStopVDSCommand, log id: 4af712bd >2012-12-17 09:44:18,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, ResetIrsVDSCommand, log id: 54857515 >2012-12-17 09:44:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61938c83] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e083abc >2012-12-17 09:44:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [61938c83] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61938c83] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e083abc >2012-12-17 09:44:19,524 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:44:19,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:44:19,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4946 >2012-12-17 09:44:19,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4946, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f0a0e88 >2012-12-17 09:44:19,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = b296419b-b531-4b5c-91b7-93fa2c518e90 >2012-12-17 09:44:20,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = b296419b-b531-4b5c-91b7-93fa2c518e90 task status = finished >2012-12-17 09:44:20,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM >2012-12-17 09:44:20,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b296419b-b531-4b5c-91b7-93fa2c518e90), log id: 30e4a457 >2012-12-17 09:44:20,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 30e4a457 >2012-12-17 09:44:20,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72e164d4, log id: 3f0a0e88 >2012-12-17 09:44:21,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:44:21,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:21,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7d6c592f >2012-12-17 09:44:21,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d6c592f >2012-12-17 09:44:21,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6146335d >2012-12-17 09:44:21,110 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:44:21,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54535cfd >2012-12-17 09:44:21,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:21,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ab4d809 >2012-12-17 09:44:21,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ab4d809 >2012-12-17 09:44:21,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 54535cfd >2012-12-17 09:44:21,130 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:44:21,157 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:44:21,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7e8b0e39 >2012-12-17 09:44:21,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:21,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73cad873 >2012-12-17 09:44:21,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73cad873 >2012-12-17 09:44:21,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7e8b0e39 >2012-12-17 09:44:21,178 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:44:21,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a48a169 >2012-12-17 09:44:21,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:21,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 28e7f452 >2012-12-17 09:44:21,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 28e7f452 >2012-12-17 09:44:21,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a48a169 >2012-12-17 09:44:21,197 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-50) [33bbf9d8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:44:29,596 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [134ba83e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:44:30,252 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [79357c7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:44:30,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ca20dda >2012-12-17 09:44:30,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c469688 >2012-12-17 09:44:30,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:30,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, SpmStopVDSCommand, log id: c469688 >2012-12-17 09:44:31,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, ResetIrsVDSCommand, log id: ca20dda >2012-12-17 09:44:31,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cf3c6ed >2012-12-17 09:44:31,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:31,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cf3c6ed >2012-12-17 09:44:41,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:44:42,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4947 >2012-12-17 09:44:42,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4947, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fb8788f >2012-12-17 09:44:42,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling started: taskId = 4d8ed956-7613-4632-b933-a0c01d114758 >2012-12-17 09:44:43,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended: taskId = 4d8ed956-7613-4632-b933-a0c01d114758 task status = finished >2012-12-17 09:44:43,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] spmStart polling ended, spm status: SPM >2012-12-17 09:44:43,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4d8ed956-7613-4632-b933-a0c01d114758), log id: 35302b57 >2012-12-17 09:44:44,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, HSMClearTaskVDSCommand, log id: 35302b57 >2012-12-17 09:44:44,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [263a433b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d6d7808, log id: 3fb8788f >2012-12-17 09:44:45,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:44:48,466 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [263a433b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:44:48,767 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [5fb78147] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:44:48,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a05d906 >2012-12-17 09:44:48,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 133754ea >2012-12-17 09:44:48,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:49,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStopVDSCommand, log id: 133754ea >2012-12-17 09:44:49,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, ResetIrsVDSCommand, log id: 1a05d906 >2012-12-17 09:44:49,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24ef34aa >2012-12-17 09:44:49,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:44:49,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24ef34aa >2012-12-17 09:44:53,366 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:44:59,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:44:59,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4948 >2012-12-17 09:44:59,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4948, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5edee38b >2012-12-17 09:44:59,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling started: taskId = 61a342a5-5d9f-4de2-87bd-56284d45e909 >2012-12-17 09:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Checking autorecoverable hosts >2012-12-17 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Autorecovering 0 hosts >2012-12-17 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Checking autorecoverable hosts done >2012-12-17 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [33cde09c] Autorecovering storage domains is disabled, skipping >2012-12-17 09:45:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling ended: taskId = 61a342a5-5d9f-4de2-87bd-56284d45e909 task status = finished >2012-12-17 09:45:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] spmStart polling ended, spm status: SPM >2012-12-17 09:45:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=61a342a5-5d9f-4de2-87bd-56284d45e909), log id: 58a9cb0d >2012-12-17 09:45:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, HSMClearTaskVDSCommand, log id: 58a9cb0d >2012-12-17 09:45:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6f4b2c75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7351eff4, log id: 5edee38b >2012-12-17 09:45:00,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:00,248 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:45:00,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55ac2e59 >2012-12-17 09:45:00,274 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6f4b2c75] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:00,367 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [16b67351] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:00,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b57bd71 >2012-12-17 09:45:00,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6680e291 >2012-12-17 09:45:00,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:00,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, SpmStopVDSCommand, log id: 6680e291 >2012-12-17 09:45:00,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, ResetIrsVDSCommand, log id: 1b57bd71 >2012-12-17 09:45:00,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [16b67351] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c1abbc2 >2012-12-17 09:45:00,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [16b67351] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:00,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [16b67351] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c1abbc2 >2012-12-17 09:45:01,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:45:01,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4949 >2012-12-17 09:45:01,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4949, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69e9f0ef >2012-12-17 09:45:01,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling started: taskId = d7aa3cd4-5784-458d-8f97-84d2a5369d1e >2012-12-17 09:45:02,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended: taskId = d7aa3cd4-5784-458d-8f97-84d2a5369d1e task status = finished >2012-12-17 09:45:02,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended, spm status: SPM >2012-12-17 09:45:02,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7aa3cd4-5784-458d-8f97-84d2a5369d1e), log id: 6d0a3c5d >2012-12-17 09:45:02,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMClearTaskVDSCommand, log id: 6d0a3c5d >2012-12-17 09:45:02,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b013598, log id: 69e9f0ef >2012-12-17 09:45:02,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:02,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:02,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: e46509f >2012-12-17 09:45:02,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: e46509f >2012-12-17 09:45:02,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 55ac2e59 >2012-12-17 09:45:02,488 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:02,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a56fe0b >2012-12-17 09:45:02,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59d3229c >2012-12-17 09:45:02,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59d3229c >2012-12-17 09:45:02,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a56fe0b >2012-12-17 09:45:02,517 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:03,653 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:45:11,308 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [7ca9a1c4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:11,618 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [61b60bdd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:11,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10c2e81a >2012-12-17 09:45:11,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e3300de >2012-12-17 09:45:11,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:11,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, SpmStopVDSCommand, log id: 5e3300de >2012-12-17 09:45:11,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, ResetIrsVDSCommand, log id: 10c2e81a >2012-12-17 09:45:11,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 342ceed0 >2012-12-17 09:45:11,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [61b60bdd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:11,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [61b60bdd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 342ceed0 >2012-12-17 09:45:13,877 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:45:21,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:45:22,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4950 >2012-12-17 09:45:22,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4950, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5fd1d016 >2012-12-17 09:45:22,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling started: taskId = 8a974492-f22b-47c3-86e1-e96d4a617a51 >2012-12-17 09:45:23,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended: taskId = 8a974492-f22b-47c3-86e1-e96d4a617a51 task status = finished >2012-12-17 09:45:23,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] spmStart polling ended, spm status: SPM >2012-12-17 09:45:23,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a974492-f22b-47c3-86e1-e96d4a617a51), log id: 2afcc4ae >2012-12-17 09:45:23,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, HSMClearTaskVDSCommand, log id: 2afcc4ae >2012-12-17 09:45:23,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [39a5c05e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38d4918e, log id: 5fd1d016 >2012-12-17 09:45:23,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:23,155 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:45:23,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5469586d >2012-12-17 09:45:23,180 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [39a5c05e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:23,270 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [3499e3ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:23,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 12ecfacd >2012-12-17 09:45:23,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67a95c79 >2012-12-17 09:45:23,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:23,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SpmStopVDSCommand, log id: 67a95c79 >2012-12-17 09:45:23,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, ResetIrsVDSCommand, log id: 12ecfacd >2012-12-17 09:45:23,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cadcac7 >2012-12-17 09:45:23,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:23,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cadcac7 >2012-12-17 09:45:23,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:45:23,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4951 >2012-12-17 09:45:23,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4951, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1373e58b >2012-12-17 09:45:23,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b >2012-12-17 09:45:24,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b task status = finished >2012-12-17 09:45:24,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM >2012-12-17 09:45:24,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b4b7627c-c1ec-45be-9ad0-9e3f96b7f53b), log id: 62444626 >2012-12-17 09:45:24,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 62444626 >2012-12-17 09:45:24,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70599b30, log id: 1373e58b >2012-12-17 09:45:24,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:24,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:24,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 69670c74 >2012-12-17 09:45:24,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69670c74 >2012-12-17 09:45:24,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5469586d >2012-12-17 09:45:24,803 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:24,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49aa4b94 >2012-12-17 09:45:24,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:24,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1700460d >2012-12-17 09:45:24,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1700460d >2012-12-17 09:45:24,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49aa4b94 >2012-12-17 09:45:24,823 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:33,703 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6e84ec96] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:33,865 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:33,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 172de3c3 >2012-12-17 09:45:33,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f7345ef >2012-12-17 09:45:33,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:34,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, SpmStopVDSCommand, log id: 5f7345ef >2012-12-17 09:45:34,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, ResetIrsVDSCommand, log id: 172de3c3 >2012-12-17 09:45:34,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6157323d >2012-12-17 09:45:34,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:34,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6157323d >2012-12-17 09:45:34,357 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:45:44,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:45:44,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4952 >2012-12-17 09:45:44,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4952, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62873213 >2012-12-17 09:45:44,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling started: taskId = ca785d31-0796-4455-b15f-c25ab9721b3a >2012-12-17 09:45:45,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling ended: taskId = ca785d31-0796-4455-b15f-c25ab9721b3a task status = finished >2012-12-17 09:45:45,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] spmStart polling ended, spm status: SPM >2012-12-17 09:45:45,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ca785d31-0796-4455-b15f-c25ab9721b3a), log id: 445bcbbf >2012-12-17 09:45:45,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, HSMClearTaskVDSCommand, log id: 445bcbbf >2012-12-17 09:45:45,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [2e298a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@669d8ba8, log id: 62873213 >2012-12-17 09:45:45,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:45,584 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:45:45,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 462ca497 >2012-12-17 09:45:45,617 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2e298a4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:47,996 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1dafbc5c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:48,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 613803f9 >2012-12-17 09:45:48,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10449a32 >2012-12-17 09:45:48,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:48,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, SpmStopVDSCommand, log id: 10449a32 >2012-12-17 09:45:48,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, ResetIrsVDSCommand, log id: 613803f9 >2012-12-17 09:45:48,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 571e80b7 >2012-12-17 09:45:48,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1dafbc5c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:48,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1dafbc5c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 571e80b7 >2012-12-17 09:45:48,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:45:48,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4953 >2012-12-17 09:45:48,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4953, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58a772ab >2012-12-17 09:45:49,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = db9f1929-2972-46a0-8d5a-aa185ff956c0 >2012-12-17 09:45:50,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = db9f1929-2972-46a0-8d5a-aa185ff956c0 task status = finished >2012-12-17 09:45:50,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM >2012-12-17 09:45:50,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db9f1929-2972-46a0-8d5a-aa185ff956c0), log id: 3338b287 >2012-12-17 09:45:50,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 3338b287 >2012-12-17 09:45:50,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f50d756, log id: 58a772ab >2012-12-17 09:45:50,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:45:50,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:50,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d6331a4 >2012-12-17 09:45:50,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d6331a4 >2012-12-17 09:45:50,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 462ca497 >2012-12-17 09:45:50,171 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:50,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b5035e2 >2012-12-17 09:45:50,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:45:50,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25d38ecb >2012-12-17 09:45:50,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25d38ecb >2012-12-17 09:45:50,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b5035e2 >2012-12-17 09:45:50,191 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:45:55,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:45:58,995 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [48ba699e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:45:59,814 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [12d72057] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:45:59,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1aea4875 >2012-12-17 09:45:59,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d5eee3e >2012-12-17 09:45:59,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:00,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, SpmStopVDSCommand, log id: d5eee3e >2012-12-17 09:46:00,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, ResetIrsVDSCommand, log id: 1aea4875 >2012-12-17 09:46:00,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [12d72057] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d8f7f8e >2012-12-17 09:46:00,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:00,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [12d72057] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d8f7f8e >2012-12-17 09:46:05,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:46:10,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:10,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4954 >2012-12-17 09:46:10,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4954, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e022704 >2012-12-17 09:46:10,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling started: taskId = d58652ea-c96b-4bd7-b975-ee8cfca18ba7 >2012-12-17 09:46:11,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended: taskId = d58652ea-c96b-4bd7-b975-ee8cfca18ba7 task status = finished >2012-12-17 09:46:11,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] spmStart polling ended, spm status: SPM >2012-12-17 09:46:11,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d58652ea-c96b-4bd7-b975-ee8cfca18ba7), log id: 5e247e4d >2012-12-17 09:46:11,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, HSMClearTaskVDSCommand, log id: 5e247e4d >2012-12-17 09:46:11,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [69834199] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2de02184, log id: 5e022704 >2012-12-17 09:46:11,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:46:12,156 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:46:12,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 623b96c0 >2012-12-17 09:46:12,195 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [69834199] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:46:12,455 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [565ac2be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:46:12,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b3bd019 >2012-12-17 09:46:12,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 667fd5cb >2012-12-17 09:46:12,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:12,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SpmStopVDSCommand, log id: 667fd5cb >2012-12-17 09:46:12,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, ResetIrsVDSCommand, log id: 5b3bd019 >2012-12-17 09:46:12,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25f1f07d >2012-12-17 09:46:12,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [565ac2be] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:12,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25f1f07d >2012-12-17 09:46:12,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:13,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4955 >2012-12-17 09:46:13,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4955, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6020028d >2012-12-17 09:46:13,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling started: taskId = 8b6a81bd-f1c1-4c13-bf97-2121e7033276 >2012-12-17 09:46:14,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling ended: taskId = 8b6a81bd-f1c1-4c13-bf97-2121e7033276 task status = finished >2012-12-17 09:46:14,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] spmStart polling ended, spm status: SPM >2012-12-17 09:46:14,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b6a81bd-f1c1-4c13-bf97-2121e7033276), log id: 6be0d07b >2012-12-17 09:46:14,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMClearTaskVDSCommand, log id: 6be0d07b >2012-12-17 09:46:14,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74977291, log id: 6020028d >2012-12-17 09:46:14,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:46:14,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:14,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7abbdd7b >2012-12-17 09:46:14,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7abbdd7b >2012-12-17 09:46:14,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 623b96c0 >2012-12-17 09:46:14,277 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:46:14,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d217121 >2012-12-17 09:46:14,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:14,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59d5be3 >2012-12-17 09:46:14,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59d5be3 >2012-12-17 09:46:14,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-39) [462eb8a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d217121 >2012-12-17 09:46:14,297 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) [462eb8a5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:46:17,435 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:46:22,967 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [4f4f0222] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:46:23,233 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [2bcb8475] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:46:23,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ddd4259 >2012-12-17 09:46:23,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1dff0cda >2012-12-17 09:46:23,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:23,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, SpmStopVDSCommand, log id: 1dff0cda >2012-12-17 09:46:23,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, ResetIrsVDSCommand, log id: 5ddd4259 >2012-12-17 09:46:23,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2243934c >2012-12-17 09:46:23,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [2bcb8475] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:23,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [2bcb8475] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2243934c >2012-12-17 09:46:27,855 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:46:33,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:33,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4956 >2012-12-17 09:46:33,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4956, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d791add >2012-12-17 09:46:33,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling started: taskId = 235c9183-bb76-4f8d-a92a-d69d02862812 >2012-12-17 09:46:34,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended: taskId = 235c9183-bb76-4f8d-a92a-d69d02862812 task status = finished >2012-12-17 09:46:34,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] spmStart polling ended, spm status: SPM >2012-12-17 09:46:34,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=235c9183-bb76-4f8d-a92a-d69d02862812), log id: 1f94cfc1 >2012-12-17 09:46:34,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, HSMClearTaskVDSCommand, log id: 1f94cfc1 >2012-12-17 09:46:34,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [786aa7bb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62d39819, log id: 1d791add >2012-12-17 09:46:34,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:46:35,061 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:46:35,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5334324f >2012-12-17 09:46:35,099 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [786aa7bb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:46:35,338 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [2353b939] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:46:35,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ed578b >2012-12-17 09:46:35,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 176eba74 >2012-12-17 09:46:35,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:35,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SpmStopVDSCommand, log id: 176eba74 >2012-12-17 09:46:35,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, ResetIrsVDSCommand, log id: ed578b >2012-12-17 09:46:35,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 957af9d >2012-12-17 09:46:35,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:35,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 957af9d >2012-12-17 09:46:35,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:35,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4957 >2012-12-17 09:46:35,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4957, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38e4e374 >2012-12-17 09:46:35,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling started: taskId = 4bf882a9-5330-4adc-87a4-29dcc13f641c >2012-12-17 09:46:36,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended: taskId = 4bf882a9-5330-4adc-87a4-29dcc13f641c task status = finished >2012-12-17 09:46:36,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended, spm status: SPM >2012-12-17 09:46:36,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4bf882a9-5330-4adc-87a4-29dcc13f641c), log id: 53f5e202 >2012-12-17 09:46:36,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMClearTaskVDSCommand, log id: 53f5e202 >2012-12-17 09:46:36,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33c91c5e, log id: 38e4e374 >2012-12-17 09:46:36,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:46:36,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:36,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f506e07 >2012-12-17 09:46:36,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f506e07 >2012-12-17 09:46:36,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5334324f >2012-12-17 09:46:36,970 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:46:36,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 59679da0 >2012-12-17 09:46:36,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:36,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5a80d926 >2012-12-17 09:46:36,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5a80d926 >2012-12-17 09:46:36,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 59679da0 >2012-12-17 09:46:36,990 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [42560f9e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:46:38,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:46:45,684 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [61938c83] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:46:47,130 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [2c358b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:46:47,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: e73ecb >2012-12-17 09:46:47,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3f37644 >2012-12-17 09:46:47,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:47,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, SpmStopVDSCommand, log id: 3f37644 >2012-12-17 09:46:47,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, ResetIrsVDSCommand, log id: e73ecb >2012-12-17 09:46:47,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bb35bc6 >2012-12-17 09:46:47,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:47,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2c358b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bb35bc6 >2012-12-17 09:46:48,418 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:46:57,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:57,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4958 >2012-12-17 09:46:57,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4958, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5be7856 >2012-12-17 09:46:57,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling started: taskId = 7f537bb5-ea61-4fcd-8164-4bf2e6f04e40 >2012-12-17 09:46:58,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling ended: taskId = 7f537bb5-ea61-4fcd-8164-4bf2e6f04e40 task status = finished >2012-12-17 09:46:58,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] spmStart polling ended, spm status: SPM >2012-12-17 09:46:58,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7f537bb5-ea61-4fcd-8164-4bf2e6f04e40), log id: 4d5d8b04 >2012-12-17 09:46:58,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, HSMClearTaskVDSCommand, log id: 4d5d8b04 >2012-12-17 09:46:58,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) [3ad3ad88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3851db4a, log id: 5be7856 >2012-12-17 09:46:58,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:46:59,032 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:46:59,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f7bcdf9 >2012-12-17 09:46:59,060 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3ad3ad88] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:46:59,192 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [7712cc49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:46:59,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a440a36 >2012-12-17 09:46:59,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3262893e >2012-12-17 09:46:59,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:59,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, SpmStopVDSCommand, log id: 3262893e >2012-12-17 09:46:59,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, ResetIrsVDSCommand, log id: 6a440a36 >2012-12-17 09:46:59,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3dd658bf >2012-12-17 09:46:59,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7712cc49] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:46:59,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7712cc49] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3dd658bf >2012-12-17 09:46:59,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:46:59,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4959 >2012-12-17 09:46:59,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4959, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69bde88 >2012-12-17 09:46:59,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = b172edb7-f8be-44dd-930e-a824187d1255 >2012-12-17 09:47:00,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = b172edb7-f8be-44dd-930e-a824187d1255 task status = finished >2012-12-17 09:47:00,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM >2012-12-17 09:47:00,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b172edb7-f8be-44dd-930e-a824187d1255), log id: 344e5305 >2012-12-17 09:47:00,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 344e5305 >2012-12-17 09:47:00,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c287adf, log id: 69bde88 >2012-12-17 09:47:00,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:47:00,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:00,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 55acb76c >2012-12-17 09:47:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55acb76c >2012-12-17 09:47:00,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f7bcdf9 >2012-12-17 09:47:00,752 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:00,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e3e3f65 >2012-12-17 09:47:00,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:00,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 389c7c14 >2012-12-17 09:47:00,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 389c7c14 >2012-12-17 09:47:00,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e3e3f65 >2012-12-17 09:47:00,773 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:08,863 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:47:09,599 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [e8edb34] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:47:09,838 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [63eed75e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:47:09,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3b43e21b >2012-12-17 09:47:10,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 692fa908 >2012-12-17 09:47:10,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:10,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, SpmStopVDSCommand, log id: 692fa908 >2012-12-17 09:47:10,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, ResetIrsVDSCommand, log id: 3b43e21b >2012-12-17 09:47:10,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e1b6a6 >2012-12-17 09:47:10,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:10,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e1b6a6 >2012-12-17 09:47:19,182 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:47:20,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:47:20,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4960 >2012-12-17 09:47:20,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4960, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 500e1121 >2012-12-17 09:47:20,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling started: taskId = 26453672-8e03-4815-9229-6aae916a2ebf >2012-12-17 09:47:21,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling ended: taskId = 26453672-8e03-4815-9229-6aae916a2ebf task status = finished >2012-12-17 09:47:21,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] spmStart polling ended, spm status: SPM >2012-12-17 09:47:21,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=26453672-8e03-4815-9229-6aae916a2ebf), log id: 76824d1c >2012-12-17 09:47:21,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, HSMClearTaskVDSCommand, log id: 76824d1c >2012-12-17 09:47:21,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [31ec2033] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6494a60c, log id: 500e1121 >2012-12-17 09:47:21,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:47:21,855 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:47:21,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 85e9fe9 >2012-12-17 09:47:21,916 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [31ec2033] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:47:22,153 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [53e9685e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:47:22,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 633d492e >2012-12-17 09:47:22,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: d011071 >2012-12-17 09:47:22,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:22,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, SpmStopVDSCommand, log id: d011071 >2012-12-17 09:47:22,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, ResetIrsVDSCommand, log id: 633d492e >2012-12-17 09:47:22,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 307cfba5 >2012-12-17 09:47:22,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:22,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [53e9685e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 307cfba5 >2012-12-17 09:47:22,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:47:22,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4961 >2012-12-17 09:47:22,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4961, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f005956 >2012-12-17 09:47:22,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 1d4eab4e-a549-4dff-8f49-db997b01ab55 >2012-12-17 09:47:23,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 1d4eab4e-a549-4dff-8f49-db997b01ab55 task status = finished >2012-12-17 09:47:23,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM >2012-12-17 09:47:23,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1d4eab4e-a549-4dff-8f49-db997b01ab55), log id: cbdc0a0 >2012-12-17 09:47:23,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: cbdc0a0 >2012-12-17 09:47:23,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7db14ee8, log id: 4f005956 >2012-12-17 09:47:23,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:47:23,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:23,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b5fdd5e >2012-12-17 09:47:23,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b5fdd5e >2012-12-17 09:47:23,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 85e9fe9 >2012-12-17 09:47:23,843 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:23,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72be752f >2012-12-17 09:47:23,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:23,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5f03daa3 >2012-12-17 09:47:23,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5f03daa3 >2012-12-17 09:47:23,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72be752f >2012-12-17 09:47:23,864 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [3e7d3817] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:29,488 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:47:32,708 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [603633ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:47:32,842 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [2b52624] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:47:32,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 30354684 >2012-12-17 09:47:32,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2951bb72 >2012-12-17 09:47:32,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:33,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, SpmStopVDSCommand, log id: 2951bb72 >2012-12-17 09:47:33,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, ResetIrsVDSCommand, log id: 30354684 >2012-12-17 09:47:33,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c029617 >2012-12-17 09:47:33,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:33,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c029617 >2012-12-17 09:47:39,708 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:47:43,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:47:43,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4962 >2012-12-17 09:47:43,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4962, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f762c0 >2012-12-17 09:47:43,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling started: taskId = 95d187dc-0da9-44ca-b393-8ed5a996d838 >2012-12-17 09:47:44,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended: taskId = 95d187dc-0da9-44ca-b393-8ed5a996d838 task status = finished >2012-12-17 09:47:44,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] spmStart polling ended, spm status: SPM >2012-12-17 09:47:44,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95d187dc-0da9-44ca-b393-8ed5a996d838), log id: 418fa6eb >2012-12-17 09:47:44,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, HSMClearTaskVDSCommand, log id: 418fa6eb >2012-12-17 09:47:44,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e17afed, log id: 1f762c0 >2012-12-17 09:47:44,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:47:44,794 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:47:44,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f36ac24 >2012-12-17 09:47:44,852 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [2d4b4fe5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:47:44,944 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [69818e3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:47:44,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fbbe20f >2012-12-17 09:47:44,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4b18c042 >2012-12-17 09:47:45,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:45,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, SpmStopVDSCommand, log id: 4b18c042 >2012-12-17 09:47:45,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, ResetIrsVDSCommand, log id: 4fbbe20f >2012-12-17 09:47:45,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79da9f0d >2012-12-17 09:47:45,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:45,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [69818e3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79da9f0d >2012-12-17 09:47:45,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:47:45,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4963 >2012-12-17 09:47:45,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4963, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dbf3ca7 >2012-12-17 09:47:45,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling started: taskId = c7f0b919-0df8-4508-904c-a8d2ecbdd29a >2012-12-17 09:47:46,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended: taskId = c7f0b919-0df8-4508-904c-a8d2ecbdd29a task status = finished >2012-12-17 09:47:46,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended, spm status: SPM >2012-12-17 09:47:46,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c7f0b919-0df8-4508-904c-a8d2ecbdd29a), log id: 5321103b >2012-12-17 09:47:46,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMClearTaskVDSCommand, log id: 5321103b >2012-12-17 09:47:46,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1fb0bc20, log id: 2dbf3ca7 >2012-12-17 09:47:46,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:47:46,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:46,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: f6d57a9 >2012-12-17 09:47:46,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: f6d57a9 >2012-12-17 09:47:46,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f36ac24 >2012-12-17 09:47:46,500 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:46,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13ff2255 >2012-12-17 09:47:46,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:46,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: ee160e >2012-12-17 09:47:46,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: ee160e >2012-12-17 09:47:46,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13ff2255 >2012-12-17 09:47:46,520 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:47:49,945 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:47:55,365 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [4d6b231f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:47:55,554 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:47:55,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2577306 >2012-12-17 09:47:55,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26ec39f3 >2012-12-17 09:47:55,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:55,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, SpmStopVDSCommand, log id: 26ec39f3 >2012-12-17 09:47:56,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, ResetIrsVDSCommand, log id: 2577306 >2012-12-17 09:47:56,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fb360cb >2012-12-17 09:47:56,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:47:56,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fb360cb >2012-12-17 09:48:00,185 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:48:07,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:07,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4964 >2012-12-17 09:48:07,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4964, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67ff327b >2012-12-17 09:48:07,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling started: taskId = 02c72871-f2d0-467a-9113-3d9a56b7246e >2012-12-17 09:48:08,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling ended: taskId = 02c72871-f2d0-467a-9113-3d9a56b7246e task status = finished >2012-12-17 09:48:08,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] spmStart polling ended, spm status: SPM >2012-12-17 09:48:08,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02c72871-f2d0-467a-9113-3d9a56b7246e), log id: 6c36358e >2012-12-17 09:48:08,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, HSMClearTaskVDSCommand, log id: 6c36358e >2012-12-17 09:48:08,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [61bf9ffc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5af63522, log id: 67ff327b >2012-12-17 09:48:08,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:08,187 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:48:08,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 70a78f58 >2012-12-17 09:48:08,217 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [61bf9ffc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:48:08,393 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [3d5285e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:48:08,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d10621 >2012-12-17 09:48:08,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bb0b606 >2012-12-17 09:48:08,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:09,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SpmStopVDSCommand, log id: 2bb0b606 >2012-12-17 09:48:09,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, ResetIrsVDSCommand, log id: 1d10621 >2012-12-17 09:48:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 519e3c94 >2012-12-17 09:48:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 519e3c94 >2012-12-17 09:48:09,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:09,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4965 >2012-12-17 09:48:09,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4965, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52abcef9 >2012-12-17 09:48:09,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling started: taskId = 1e24d303-bbfe-474b-86d4-022bac2afb3c >2012-12-17 09:48:10,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling ended: taskId = 1e24d303-bbfe-474b-86d4-022bac2afb3c task status = finished >2012-12-17 09:48:10,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] spmStart polling ended, spm status: SPM >2012-12-17 09:48:10,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1e24d303-bbfe-474b-86d4-022bac2afb3c), log id: 59aa1661 >2012-12-17 09:48:10,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMClearTaskVDSCommand, log id: 59aa1661 >2012-12-17 09:48:10,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2da36844, log id: 52abcef9 >2012-12-17 09:48:10,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:10,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:10,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6d69d974 >2012-12-17 09:48:10,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d69d974 >2012-12-17 09:48:10,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 70a78f58 >2012-12-17 09:48:10,293 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:48:10,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 404386ca >2012-12-17 09:48:10,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:10,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 182129c1 >2012-12-17 09:48:10,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 182129c1 >2012-12-17 09:48:10,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [515f648d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 404386ca >2012-12-17 09:48:10,336 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [515f648d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:48:10,425 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:48:19,181 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [7781d588] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:48:19,325 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [469720a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:48:19,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f91495 >2012-12-17 09:48:19,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75975b4f >2012-12-17 09:48:19,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:19,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, SpmStopVDSCommand, log id: 75975b4f >2012-12-17 09:48:19,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, ResetIrsVDSCommand, log id: 48f91495 >2012-12-17 09:48:19,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [469720a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61a7eee9 >2012-12-17 09:48:19,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:19,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [469720a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61a7eee9 >2012-12-17 09:48:20,601 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:48:29,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:29,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4966 >2012-12-17 09:48:29,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4966, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32bc1a2d >2012-12-17 09:48:29,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling started: taskId = 7e06ddf1-4d36-4ecd-9d80-323272d41594 >2012-12-17 09:48:30,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended: taskId = 7e06ddf1-4d36-4ecd-9d80-323272d41594 task status = finished >2012-12-17 09:48:30,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] spmStart polling ended, spm status: SPM >2012-12-17 09:48:30,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e06ddf1-4d36-4ecd-9d80-323272d41594), log id: 5b074175 >2012-12-17 09:48:30,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, HSMClearTaskVDSCommand, log id: 5b074175 >2012-12-17 09:48:30,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [41718aec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34153ba7, log id: 32bc1a2d >2012-12-17 09:48:31,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:31,738 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:48:31,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dbea72 >2012-12-17 09:48:31,750 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [41718aec] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:48:31,902 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:48:31,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c25fa0d >2012-12-17 09:48:31,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 381760d5 >2012-12-17 09:48:31,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:32,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStopVDSCommand, log id: 381760d5 >2012-12-17 09:48:32,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, ResetIrsVDSCommand, log id: 7c25fa0d >2012-12-17 09:48:32,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 228a71b4 >2012-12-17 09:48:32,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:32,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 228a71b4 >2012-12-17 09:48:32,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:32,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4967 >2012-12-17 09:48:32,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4967, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 610a2666 >2012-12-17 09:48:32,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 5ea942f9-9608-4d4f-abd4-1b7d89710683 >2012-12-17 09:48:33,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 5ea942f9-9608-4d4f-abd4-1b7d89710683 task status = finished >2012-12-17 09:48:33,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM >2012-12-17 09:48:33,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ea942f9-9608-4d4f-abd4-1b7d89710683), log id: 4ddcaf9f >2012-12-17 09:48:33,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 4ddcaf9f >2012-12-17 09:48:33,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a813717, log id: 610a2666 >2012-12-17 09:48:33,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:33,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:33,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 38bbd0a9 >2012-12-17 09:48:33,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 38bbd0a9 >2012-12-17 09:48:33,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dbea72 >2012-12-17 09:48:33,446 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:48:33,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ad2274e >2012-12-17 09:48:33,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:33,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 785fad50 >2012-12-17 09:48:33,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 785fad50 >2012-12-17 09:48:33,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ad2274e >2012-12-17 09:48:33,467 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:48:42,028 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:48:42,305 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [7ac3ac7b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:48:42,518 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [49d29b47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:48:42,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 684337fe >2012-12-17 09:48:42,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60fa540b >2012-12-17 09:48:42,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:42,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, SpmStopVDSCommand, log id: 60fa540b >2012-12-17 09:48:42,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, ResetIrsVDSCommand, log id: 684337fe >2012-12-17 09:48:42,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ae86858 >2012-12-17 09:48:42,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [49d29b47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:42,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [49d29b47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ae86858 >2012-12-17 09:48:52,209 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:48:52,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:52,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4968 >2012-12-17 09:48:52,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4968, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7838e427 >2012-12-17 09:48:52,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling started: taskId = 72a8c156-96d2-4e80-a71a-0f61f23ecd82 >2012-12-17 09:48:53,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended: taskId = 72a8c156-96d2-4e80-a71a-0f61f23ecd82 task status = finished >2012-12-17 09:48:53,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] spmStart polling ended, spm status: SPM >2012-12-17 09:48:53,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72a8c156-96d2-4e80-a71a-0f61f23ecd82), log id: 60e0d390 >2012-12-17 09:48:54,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, HSMClearTaskVDSCommand, log id: 60e0d390 >2012-12-17 09:48:54,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1ab078c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4861c4a0, log id: 7838e427 >2012-12-17 09:48:54,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:54,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:48:54,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5c410c4a >2012-12-17 09:48:54,092 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1ab078c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:48:54,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [9d8cf45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:48:54,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: c91a391 >2012-12-17 09:48:54,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b37cb5f >2012-12-17 09:48:54,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:54,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, SpmStopVDSCommand, log id: b37cb5f >2012-12-17 09:48:54,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, ResetIrsVDSCommand, log id: c91a391 >2012-12-17 09:48:54,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a82e070 >2012-12-17 09:48:54,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:54,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a82e070 >2012-12-17 09:48:54,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:48:54,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4969 >2012-12-17 09:48:54,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4969, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: dc5c9ad >2012-12-17 09:48:54,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7 >2012-12-17 09:48:55,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7 task status = finished >2012-12-17 09:48:55,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM >2012-12-17 09:48:55,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b4bc3b5d-6605-4dba-b706-ceb2fa52b1d7), log id: 71ab63ea >2012-12-17 09:48:56,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 71ab63ea >2012-12-17 09:48:56,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b2cd1f3, log id: dc5c9ad >2012-12-17 09:48:56,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:48:56,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:56,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: dc5ee60 >2012-12-17 09:48:56,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: dc5ee60 >2012-12-17 09:48:56,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5c410c4a >2012-12-17 09:48:56,460 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:48:56,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3da4e7b0 >2012-12-17 09:48:56,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:48:56,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51c04146 >2012-12-17 09:48:56,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51c04146 >2012-12-17 09:48:56,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3da4e7b0 >2012-12-17 09:48:56,481 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:49:02,428 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:49:04,938 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [680acadd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:49:05,090 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-29) [2b3eea84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:49:05,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 671c05fc >2012-12-17 09:49:05,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3578ebbc >2012-12-17 09:49:05,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:05,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SpmStopVDSCommand, log id: 3578ebbc >2012-12-17 09:49:06,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, ResetIrsVDSCommand, log id: 671c05fc >2012-12-17 09:49:06,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e552012 >2012-12-17 09:49:06,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:06,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e552012 >2012-12-17 09:49:12,652 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:49:16,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:49:16,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4970 >2012-12-17 09:49:16,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4970, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7aa26cf6 >2012-12-17 09:49:16,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling started: taskId = aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528 >2012-12-17 09:49:17,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended: taskId = aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528 task status = finished >2012-12-17 09:49:17,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] spmStart polling ended, spm status: SPM >2012-12-17 09:49:17,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aa5ded6e-55d8-42b9-b6a1-ac14fa5fa528), log id: 5f65b4b6 >2012-12-17 09:49:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, HSMClearTaskVDSCommand, log id: 5f65b4b6 >2012-12-17 09:49:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2c554bbe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e3b49a8, log id: 7aa26cf6 >2012-12-17 09:49:17,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:49:17,545 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:49:17,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 438204a8 >2012-12-17 09:49:17,573 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2c554bbe] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:49:17,660 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [510758b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:49:17,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6df764f2 >2012-12-17 09:49:17,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2fcf2b3 >2012-12-17 09:49:17,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:17,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, SpmStopVDSCommand, log id: 2fcf2b3 >2012-12-17 09:49:17,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, ResetIrsVDSCommand, log id: 6df764f2 >2012-12-17 09:49:17,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [510758b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e67df6c >2012-12-17 09:49:17,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [510758b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:17,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [510758b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e67df6c >2012-12-17 09:49:17,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:49:17,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4971 >2012-12-17 09:49:17,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4971, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2644c0e7 >2012-12-17 09:49:17,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling started: taskId = 2172aef2-2d65-46ca-8e02-20cd56b19bee >2012-12-17 09:49:19,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended: taskId = 2172aef2-2d65-46ca-8e02-20cd56b19bee task status = finished >2012-12-17 09:49:19,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended, spm status: SPM >2012-12-17 09:49:19,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2172aef2-2d65-46ca-8e02-20cd56b19bee), log id: 5943be32 >2012-12-17 09:49:19,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMClearTaskVDSCommand, log id: 5943be32 >2012-12-17 09:49:19,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e7bf266, log id: 2644c0e7 >2012-12-17 09:49:19,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:49:19,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:19,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 19049ecb >2012-12-17 09:49:19,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 19049ecb >2012-12-17 09:49:19,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 438204a8 >2012-12-17 09:49:19,137 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:49:19,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a4fee33 >2012-12-17 09:49:19,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:19,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4abd8f52 >2012-12-17 09:49:19,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4abd8f52 >2012-12-17 09:49:19,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a4fee33 >2012-12-17 09:49:19,157 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-86) [56a7c4a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:49:22,853 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:49:28,000 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [65ba9edd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:49:28,136 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [4bbb254] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:49:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 61bce2b0 >2012-12-17 09:49:28,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a2186ff >2012-12-17 09:49:28,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:28,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, SpmStopVDSCommand, log id: 5a2186ff >2012-12-17 09:49:28,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, ResetIrsVDSCommand, log id: 61bce2b0 >2012-12-17 09:49:28,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 340e73ba >2012-12-17 09:49:28,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4bbb254] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:28,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [4bbb254] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 340e73ba >2012-12-17 09:49:33,091 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:49:38,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:49:38,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4972 >2012-12-17 09:49:38,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4972, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7bb857ed >2012-12-17 09:49:38,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling started: taskId = 7a86b9e4-ee9b-4f79-be93-b35e18102e0e >2012-12-17 09:49:39,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling ended: taskId = 7a86b9e4-ee9b-4f79-be93-b35e18102e0e task status = finished >2012-12-17 09:49:39,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] spmStart polling ended, spm status: SPM >2012-12-17 09:49:39,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7a86b9e4-ee9b-4f79-be93-b35e18102e0e), log id: 6709929b >2012-12-17 09:49:39,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, HSMClearTaskVDSCommand, log id: 6709929b >2012-12-17 09:49:39,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [58bd42f7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@257832f6, log id: 7bb857ed >2012-12-17 09:49:39,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:49:39,908 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:49:39,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ef4a17b >2012-12-17 09:49:39,908 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [58bd42f7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:49:40,110 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [38047437] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:49:40,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7c215e40 >2012-12-17 09:49:40,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35021459 >2012-12-17 09:49:40,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:40,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, SpmStopVDSCommand, log id: 35021459 >2012-12-17 09:49:40,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, ResetIrsVDSCommand, log id: 7c215e40 >2012-12-17 09:49:40,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [38047437] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e1c52dd >2012-12-17 09:49:40,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [38047437] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:40,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [38047437] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e1c52dd >2012-12-17 09:49:40,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:49:41,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4973 >2012-12-17 09:49:41,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4973, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b822200 >2012-12-17 09:49:41,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling started: taskId = 46b1d296-4158-41a4-9ac2-1d2468c3326f >2012-12-17 09:49:42,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended: taskId = 46b1d296-4158-41a4-9ac2-1d2468c3326f task status = finished >2012-12-17 09:49:42,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended, spm status: SPM >2012-12-17 09:49:42,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=46b1d296-4158-41a4-9ac2-1d2468c3326f), log id: 30e94a07 >2012-12-17 09:49:42,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMClearTaskVDSCommand, log id: 30e94a07 >2012-12-17 09:49:42,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23fb2aa7, log id: 6b822200 >2012-12-17 09:49:42,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:49:42,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:42,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2a815e1d >2012-12-17 09:49:42,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2a815e1d >2012-12-17 09:49:42,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ef4a17b >2012-12-17 09:49:42,555 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:49:42,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1349d7b8 >2012-12-17 09:49:42,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:42,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25ec675e >2012-12-17 09:49:42,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25ec675e >2012-12-17 09:49:42,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1349d7b8 >2012-12-17 09:49:42,575 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) [75733504] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:49:43,650 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:49:51,394 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [130592b9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:49:51,513 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [312ad09c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:49:51,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4182e797 >2012-12-17 09:49:51,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 275da8bd >2012-12-17 09:49:51,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:51,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStopVDSCommand, log id: 275da8bd >2012-12-17 09:49:51,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, ResetIrsVDSCommand, log id: 4182e797 >2012-12-17 09:49:51,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 567f91e8 >2012-12-17 09:49:51,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:49:51,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 567f91e8 >2012-12-17 09:49:53,837 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Checking autorecoverable hosts >2012-12-17 09:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Autorecovering 0 hosts >2012-12-17 09:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Checking autorecoverable hosts done >2012-12-17 09:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [12d72057] Autorecovering storage domains is disabled, skipping >2012-12-17 09:50:01,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:50:04,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4974 >2012-12-17 09:50:04,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4974, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 487299e4 >2012-12-17 09:50:04,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling started: taskId = 03efe095-7f88-49fa-a5c3-e1a0c05d7efa >2012-12-17 09:50:05,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling ended: taskId = 03efe095-7f88-49fa-a5c3-e1a0c05d7efa task status = finished >2012-12-17 09:50:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] spmStart polling ended, spm status: SPM >2012-12-17 09:50:05,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=03efe095-7f88-49fa-a5c3-e1a0c05d7efa), log id: 4de126b3 >2012-12-17 09:50:05,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, HSMClearTaskVDSCommand, log id: 4de126b3 >2012-12-17 09:50:05,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [519a0bb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b3d9b6, log id: 487299e4 >2012-12-17 09:50:05,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:50:05,284 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:50:05,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2df299ca >2012-12-17 09:50:05,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [519a0bb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:50:05,464 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2111ab39] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:50:05,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45276ffc >2012-12-17 09:50:05,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5b38696b >2012-12-17 09:50:05,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:06,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, SpmStopVDSCommand, log id: 5b38696b >2012-12-17 09:50:06,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, ResetIrsVDSCommand, log id: 45276ffc >2012-12-17 09:50:06,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24803349 >2012-12-17 09:50:06,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:06,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24803349 >2012-12-17 09:50:06,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:50:06,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4975 >2012-12-17 09:50:06,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4975, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e0ee927 >2012-12-17 09:50:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling started: taskId = 570eccbd-1961-475c-aefe-8c719d723668 >2012-12-17 09:50:07,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended: taskId = 570eccbd-1961-475c-aefe-8c719d723668 task status = finished >2012-12-17 09:50:07,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended, spm status: SPM >2012-12-17 09:50:07,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=570eccbd-1961-475c-aefe-8c719d723668), log id: 5fbf22b9 >2012-12-17 09:50:07,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMClearTaskVDSCommand, log id: 5fbf22b9 >2012-12-17 09:50:07,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d80d0ca, log id: e0ee927 >2012-12-17 09:50:08,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:50:08,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:08,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 181203e3 >2012-12-17 09:50:08,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 181203e3 >2012-12-17 09:50:08,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2df299ca >2012-12-17 09:50:08,047 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:50:08,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46bd386d >2012-12-17 09:50:08,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:08,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 33ec1547 >2012-12-17 09:50:08,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33ec1547 >2012-12-17 09:50:08,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46bd386d >2012-12-17 09:50:08,068 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-7) [13c8ba9c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:50:14,649 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:50:16,921 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [49d29b47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:50:17,042 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [3e0f3441] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:50:17,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b26b838 >2012-12-17 09:50:17,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 26618845 >2012-12-17 09:50:17,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:17,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStopVDSCommand, log id: 26618845 >2012-12-17 09:50:17,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, ResetIrsVDSCommand, log id: 1b26b838 >2012-12-17 09:50:17,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 267254c3 >2012-12-17 09:50:17,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:17,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 267254c3 >2012-12-17 09:50:25,203 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:50:27,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:50:27,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4976 >2012-12-17 09:50:27,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4976, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5631fdc9 >2012-12-17 09:50:27,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling started: taskId = dc8b7586-3c66-46a8-a8a6-19fa64aef5cb >2012-12-17 09:50:28,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling ended: taskId = dc8b7586-3c66-46a8-a8a6-19fa64aef5cb task status = finished >2012-12-17 09:50:28,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] spmStart polling ended, spm status: SPM >2012-12-17 09:50:28,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [be98275] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc8b7586-3c66-46a8-a8a6-19fa64aef5cb), log id: 1e7d9cb3 >2012-12-17 09:50:28,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, HSMClearTaskVDSCommand, log id: 1e7d9cb3 >2012-12-17 09:50:28,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [be98275] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48babb68, log id: 5631fdc9 >2012-12-17 09:50:28,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:50:28,802 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:50:28,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 338ea258 >2012-12-17 09:50:28,849 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [be98275] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:50:29,296 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [5e506f3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:50:30,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3390c1b4 >2012-12-17 09:50:30,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 538d09eb >2012-12-17 09:50:30,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:30,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, SpmStopVDSCommand, log id: 538d09eb >2012-12-17 09:50:31,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, ResetIrsVDSCommand, log id: 3390c1b4 >2012-12-17 09:50:31,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c967043 >2012-12-17 09:50:31,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:31,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c967043 >2012-12-17 09:50:33,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:50:33,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4977 >2012-12-17 09:50:33,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4977, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b4efdc >2012-12-17 09:50:33,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling started: taskId = 141603a7-d0f8-41f9-bb3a-30ada5c07a1f >2012-12-17 09:50:34,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended: taskId = 141603a7-d0f8-41f9-bb3a-30ada5c07a1f task status = finished >2012-12-17 09:50:34,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended, spm status: SPM >2012-12-17 09:50:34,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=141603a7-d0f8-41f9-bb3a-30ada5c07a1f), log id: 3f47b1b1 >2012-12-17 09:50:34,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMClearTaskVDSCommand, log id: 3f47b1b1 >2012-12-17 09:50:34,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7be2fbeb, log id: 4b4efdc >2012-12-17 09:50:34,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:50:34,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:34,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 11d2420c >2012-12-17 09:50:34,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11d2420c >2012-12-17 09:50:34,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 338ea258 >2012-12-17 09:50:34,527 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:50:34,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62470148 >2012-12-17 09:50:34,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:34,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bc51bd6 >2012-12-17 09:50:34,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bc51bd6 >2012-12-17 09:50:34,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 62470148 >2012-12-17 09:50:34,548 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:50:35,542 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:50:43,842 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [278943a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:50:44,433 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [7fd67d3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:50:44,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f9bdcc1 >2012-12-17 09:50:44,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 45b52c72 >2012-12-17 09:50:44,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:44,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, SpmStopVDSCommand, log id: 45b52c72 >2012-12-17 09:50:45,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, ResetIrsVDSCommand, log id: 4f9bdcc1 >2012-12-17 09:50:45,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 482e2e92 >2012-12-17 09:50:45,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7fd67d3e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:45,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7fd67d3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 482e2e92 >2012-12-17 09:50:47,027 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:50:55,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:50:55,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4978 >2012-12-17 09:50:55,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4978, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71fcbe3 >2012-12-17 09:50:55,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling started: taskId = ed15f365-6f0c-4983-8a2f-e34c91774a17 >2012-12-17 09:50:56,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling ended: taskId = ed15f365-6f0c-4983-8a2f-e34c91774a17 task status = finished >2012-12-17 09:50:56,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] spmStart polling ended, spm status: SPM >2012-12-17 09:50:56,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ed15f365-6f0c-4983-8a2f-e34c91774a17), log id: 48e03ce3 >2012-12-17 09:50:56,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, HSMClearTaskVDSCommand, log id: 48e03ce3 >2012-12-17 09:50:56,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [7fbbf72c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@79bd8075, log id: 71fcbe3 >2012-12-17 09:50:56,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:50:57,440 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:50:57,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e2ab496 >2012-12-17 09:50:57,468 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7fbbf72c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:50:57,906 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [477daf1e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:50:59,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24bdfadb >2012-12-17 09:50:59,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5937ffac >2012-12-17 09:50:59,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:50:59,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SpmStopVDSCommand, log id: 5937ffac >2012-12-17 09:50:59,611 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:51:00,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, ResetIrsVDSCommand, log id: 24bdfadb >2012-12-17 09:51:00,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4340bfd >2012-12-17 09:51:00,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:00,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4340bfd >2012-12-17 09:51:03,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:51:03,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4979 >2012-12-17 09:51:03,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4979, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28aeda01 >2012-12-17 09:51:03,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling started: taskId = 45cf1767-48e9-4dba-a2ab-b42afc9a8edc >2012-12-17 09:51:04,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended: taskId = 45cf1767-48e9-4dba-a2ab-b42afc9a8edc task status = finished >2012-12-17 09:51:04,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended, spm status: SPM >2012-12-17 09:51:04,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=45cf1767-48e9-4dba-a2ab-b42afc9a8edc), log id: 44493f14 >2012-12-17 09:51:04,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMClearTaskVDSCommand, log id: 44493f14 >2012-12-17 09:51:04,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24cc5d12, log id: 28aeda01 >2012-12-17 09:51:04,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:51:05,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:05,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ff80de4 >2012-12-17 09:51:05,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ff80de4 >2012-12-17 09:51:05,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e2ab496 >2012-12-17 09:51:05,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:05,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7658355e >2012-12-17 09:51:05,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:05,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 34795f86 >2012-12-17 09:51:05,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 34795f86 >2012-12-17 09:51:05,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7658355e >2012-12-17 09:51:05,091 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [7a6ed577] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:05,137 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:51:05,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 26d2482c >2012-12-17 09:51:05,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:05,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f45ee5f >2012-12-17 09:51:05,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f45ee5f >2012-12-17 09:51:05,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 26d2482c >2012-12-17 09:51:05,157 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:05,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5824d80b >2012-12-17 09:51:05,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:05,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1450ee3d >2012-12-17 09:51:05,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1450ee3d >2012-12-17 09:51:05,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5824d80b >2012-12-17 09:51:05,177 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:13,845 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [632afe2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:51:14,700 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [23663da4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:51:14,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6879d3e0 >2012-12-17 09:51:14,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6bba85a1 >2012-12-17 09:51:14,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:15,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, SpmStopVDSCommand, log id: 6bba85a1 >2012-12-17 09:51:15,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, ResetIrsVDSCommand, log id: 6879d3e0 >2012-12-17 09:51:15,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e65d6c3 >2012-12-17 09:51:15,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:15,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e65d6c3 >2012-12-17 09:51:23,392 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:51:25,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:51:25,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4980 >2012-12-17 09:51:25,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4980, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5265faaf >2012-12-17 09:51:25,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling started: taskId = 5a62e718-7775-4eb5-9a32-eb7de347cb52 >2012-12-17 09:51:26,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling ended: taskId = 5a62e718-7775-4eb5-9a32-eb7de347cb52 task status = finished >2012-12-17 09:51:26,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] spmStart polling ended, spm status: SPM >2012-12-17 09:51:26,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a62e718-7775-4eb5-9a32-eb7de347cb52), log id: 405dad8c >2012-12-17 09:51:26,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, HSMClearTaskVDSCommand, log id: 405dad8c >2012-12-17 09:51:26,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@462ce357, log id: 5265faaf >2012-12-17 09:51:26,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:51:27,429 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:51:27,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f241436 >2012-12-17 09:51:27,511 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1fe1ed7c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:51:27,958 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [405e01] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:51:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 10505760 >2012-12-17 09:51:28,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 38fe75c3 >2012-12-17 09:51:28,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:28,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, SpmStopVDSCommand, log id: 38fe75c3 >2012-12-17 09:51:28,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, ResetIrsVDSCommand, log id: 10505760 >2012-12-17 09:51:28,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [405e01] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 418de0b7 >2012-12-17 09:51:28,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:28,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [405e01] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 418de0b7 >2012-12-17 09:51:28,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:51:28,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4981 >2012-12-17 09:51:28,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4981, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35ac9919 >2012-12-17 09:51:28,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling started: taskId = 25063552-b177-43fc-a41a-aa8e973fc67f >2012-12-17 09:51:29,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended: taskId = 25063552-b177-43fc-a41a-aa8e973fc67f task status = finished >2012-12-17 09:51:29,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] spmStart polling ended, spm status: SPM >2012-12-17 09:51:29,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=25063552-b177-43fc-a41a-aa8e973fc67f), log id: 19a1fed5 >2012-12-17 09:51:29,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMClearTaskVDSCommand, log id: 19a1fed5 >2012-12-17 09:51:29,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a615f18, log id: 35ac9919 >2012-12-17 09:51:29,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:51:29,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:29,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 195b06b6 >2012-12-17 09:51:29,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 195b06b6 >2012-12-17 09:51:29,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f241436 >2012-12-17 09:51:29,851 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:29,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2715a3bb >2012-12-17 09:51:29,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:29,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4945fbba >2012-12-17 09:51:29,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4945fbba >2012-12-17 09:51:29,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-90) [4ea30a59] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2715a3bb >2012-12-17 09:51:29,871 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-90) [4ea30a59] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:33,991 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:51:38,751 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1eb0b032] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:51:39,233 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [854c8c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:51:39,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e0020c1 >2012-12-17 09:51:39,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 499dbf8 >2012-12-17 09:51:39,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:39,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, SpmStopVDSCommand, log id: 499dbf8 >2012-12-17 09:51:39,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, ResetIrsVDSCommand, log id: 6e0020c1 >2012-12-17 09:51:39,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6def32c8 >2012-12-17 09:51:39,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:39,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6def32c8 >2012-12-17 09:51:44,373 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:51:50,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:51:50,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4982 >2012-12-17 09:51:50,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4982, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ed879bd >2012-12-17 09:51:50,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling started: taskId = 5a0019bc-b4d8-41aa-91db-ed9e752f3093 >2012-12-17 09:51:51,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended: taskId = 5a0019bc-b4d8-41aa-91db-ed9e752f3093 task status = finished >2012-12-17 09:51:51,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] spmStart polling ended, spm status: SPM >2012-12-17 09:51:51,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a0019bc-b4d8-41aa-91db-ed9e752f3093), log id: 426300be >2012-12-17 09:51:51,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, HSMClearTaskVDSCommand, log id: 426300be >2012-12-17 09:51:51,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [13c8ba9c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71c5dbfe, log id: 7ed879bd >2012-12-17 09:51:51,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:51:51,758 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:51:51,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4cfe4f7d >2012-12-17 09:51:51,810 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [13c8ba9c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:51:52,140 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [4f15776f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:51:52,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b98d819 >2012-12-17 09:51:52,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 68e2bca >2012-12-17 09:51:52,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:52,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, SpmStopVDSCommand, log id: 68e2bca >2012-12-17 09:51:52,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, ResetIrsVDSCommand, log id: b98d819 >2012-12-17 09:51:52,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c3ae027 >2012-12-17 09:51:52,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4f15776f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:52,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [4f15776f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c3ae027 >2012-12-17 09:51:52,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:51:52,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4983 >2012-12-17 09:51:52,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4983, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 24b61521 >2012-12-17 09:51:52,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling started: taskId = 5c0adc66-acaa-4264-a331-763d78e2af57 >2012-12-17 09:51:53,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling ended: taskId = 5c0adc66-acaa-4264-a331-763d78e2af57 task status = finished >2012-12-17 09:51:53,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] spmStart polling ended, spm status: SPM >2012-12-17 09:51:53,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5c0adc66-acaa-4264-a331-763d78e2af57), log id: 4a992b09 >2012-12-17 09:51:53,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMClearTaskVDSCommand, log id: 4a992b09 >2012-12-17 09:51:53,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@dd88222, log id: 24b61521 >2012-12-17 09:51:53,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:51:53,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:53,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1defc929 >2012-12-17 09:51:53,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1defc929 >2012-12-17 09:51:53,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4cfe4f7d >2012-12-17 09:51:53,948 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:53,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 697fb219 >2012-12-17 09:51:53,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:51:53,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 291c02cc >2012-12-17 09:51:53,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 291c02cc >2012-12-17 09:51:53,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [477daf1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 697fb219 >2012-12-17 09:51:53,967 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [477daf1e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:51:54,802 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:52:02,728 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [78e614fb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:52:03,197 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [53c2bb48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:52:03,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4eed7a97 >2012-12-17 09:52:03,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 58be88d2 >2012-12-17 09:52:03,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:03,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, SpmStopVDSCommand, log id: 58be88d2 >2012-12-17 09:52:03,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, ResetIrsVDSCommand, log id: 4eed7a97 >2012-12-17 09:52:03,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40d3e26 >2012-12-17 09:52:03,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [53c2bb48] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:03,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [53c2bb48] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40d3e26 >2012-12-17 09:52:05,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:52:13,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:52:13,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4984 >2012-12-17 09:52:13,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4984, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67388b14 >2012-12-17 09:52:13,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling started: taskId = e7b898fa-052d-40d0-9c16-46ba5095fc5c >2012-12-17 09:52:14,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling ended: taskId = e7b898fa-052d-40d0-9c16-46ba5095fc5c task status = finished >2012-12-17 09:52:15,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] spmStart polling ended, spm status: SPM >2012-12-17 09:52:15,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7b898fa-052d-40d0-9c16-46ba5095fc5c), log id: 2851bf87 >2012-12-17 09:52:15,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, HSMClearTaskVDSCommand, log id: 2851bf87 >2012-12-17 09:52:15,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [9d8cf45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@ef54cbf, log id: 67388b14 >2012-12-17 09:52:15,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:52:15,283 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:52:15,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34d5b0f0 >2012-12-17 09:52:15,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [9d8cf45] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:52:16,906 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [37d03d47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:52:17,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bb36d4f >2012-12-17 09:52:17,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3730d4c7 >2012-12-17 09:52:17,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SpmStopVDSCommand, log id: 3730d4c7 >2012-12-17 09:52:17,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, ResetIrsVDSCommand, log id: 5bb36d4f >2012-12-17 09:52:17,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6cec7f0a >2012-12-17 09:52:17,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:17,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6cec7f0a >2012-12-17 09:52:17,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:52:17,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4985 >2012-12-17 09:52:17,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4985, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b26b2e1 >2012-12-17 09:52:17,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = 5f5cad79-7cb0-4fde-88b7-9209f448e794 >2012-12-17 09:52:18,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = 5f5cad79-7cb0-4fde-88b7-9209f448e794 task status = finished >2012-12-17 09:52:18,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM >2012-12-17 09:52:18,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5f5cad79-7cb0-4fde-88b7-9209f448e794), log id: 34be4ba4 >2012-12-17 09:52:18,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 34be4ba4 >2012-12-17 09:52:18,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@39581709, log id: 5b26b2e1 >2012-12-17 09:52:18,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:52:19,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:19,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 277eb0bf >2012-12-17 09:52:19,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 277eb0bf >2012-12-17 09:52:19,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34d5b0f0 >2012-12-17 09:52:19,090 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:52:19,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e85dfda >2012-12-17 09:52:19,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:19,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 885ba77 >2012-12-17 09:52:19,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 885ba77 >2012-12-17 09:52:19,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e85dfda >2012-12-17 09:52:19,110 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:52:27,168 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:52:27,801 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7fd67d3e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:52:28,122 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [5b398282] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:52:28,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e6b7435 >2012-12-17 09:52:28,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62535698 >2012-12-17 09:52:28,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:28,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, SpmStopVDSCommand, log id: 62535698 >2012-12-17 09:52:28,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, ResetIrsVDSCommand, log id: 6e6b7435 >2012-12-17 09:52:28,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69d5eb71 >2012-12-17 09:52:28,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:28,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69d5eb71 >2012-12-17 09:52:37,475 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:52:38,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:52:38,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4986 >2012-12-17 09:52:38,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4986, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bc61824 >2012-12-17 09:52:38,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling started: taskId = d74845c2-2b1b-46d9-8ea1-3eb6382650e6 >2012-12-17 09:52:39,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling ended: taskId = d74845c2-2b1b-46d9-8ea1-3eb6382650e6 task status = finished >2012-12-17 09:52:39,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] spmStart polling ended, spm status: SPM >2012-12-17 09:52:39,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d74845c2-2b1b-46d9-8ea1-3eb6382650e6), log id: 68270543 >2012-12-17 09:52:39,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, HSMClearTaskVDSCommand, log id: 68270543 >2012-12-17 09:52:39,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2111ab39] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59caa720, log id: 6bc61824 >2012-12-17 09:52:39,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:52:40,061 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:52:40,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4dfdbd3d >2012-12-17 09:52:40,135 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2111ab39] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:52:40,525 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2b017a7d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:52:41,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5f40a586 >2012-12-17 09:52:41,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1574aabc >2012-12-17 09:52:41,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:42,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStopVDSCommand, log id: 1574aabc >2012-12-17 09:52:42,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, ResetIrsVDSCommand, log id: 5f40a586 >2012-12-17 09:52:42,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ec9d908 >2012-12-17 09:52:42,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:42,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ec9d908 >2012-12-17 09:52:42,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:52:42,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4987 >2012-12-17 09:52:42,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4987, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f1c2781 >2012-12-17 09:52:42,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling started: taskId = e3477ac1-c1b7-4483-b115-c0bb4a42aca4 >2012-12-17 09:52:43,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling ended: taskId = e3477ac1-c1b7-4483-b115-c0bb4a42aca4 task status = finished >2012-12-17 09:52:43,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] spmStart polling ended, spm status: SPM >2012-12-17 09:52:43,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e3477ac1-c1b7-4483-b115-c0bb4a42aca4), log id: 33138b41 >2012-12-17 09:52:43,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMClearTaskVDSCommand, log id: 33138b41 >2012-12-17 09:52:43,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@481b7299, log id: 7f1c2781 >2012-12-17 09:52:43,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:52:43,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:43,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24578e7f >2012-12-17 09:52:43,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24578e7f >2012-12-17 09:52:43,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4dfdbd3d >2012-12-17 09:52:43,770 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:52:43,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 704c467c >2012-12-17 09:52:43,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:43,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7f1b5bcf >2012-12-17 09:52:43,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f1b5bcf >2012-12-17 09:52:43,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [3499e3ae] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 704c467c >2012-12-17 09:52:43,791 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [3499e3ae] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:52:47,751 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:52:52,578 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [53c2bb48] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:52:53,021 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [54d6e7f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:52:53,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 447f2569 >2012-12-17 09:52:53,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 34a61f40 >2012-12-17 09:52:53,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:53,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, SpmStopVDSCommand, log id: 34a61f40 >2012-12-17 09:52:53,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, ResetIrsVDSCommand, log id: 447f2569 >2012-12-17 09:52:53,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2035f8ff >2012-12-17 09:52:53,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [54d6e7f6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:52:53,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [54d6e7f6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2035f8ff >2012-12-17 09:52:58,395 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:53:03,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:03,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4988 >2012-12-17 09:53:03,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4988, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e8843a8 >2012-12-17 09:53:03,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling started: taskId = d0ef27b0-b4a8-4bbe-804b-7c5452423ea5 >2012-12-17 09:53:04,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling ended: taskId = d0ef27b0-b4a8-4bbe-804b-7c5452423ea5 task status = finished >2012-12-17 09:53:04,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] spmStart polling ended, spm status: SPM >2012-12-17 09:53:04,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d0ef27b0-b4a8-4bbe-804b-7c5452423ea5), log id: 2978a13c >2012-12-17 09:53:04,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, HSMClearTaskVDSCommand, log id: 2978a13c >2012-12-17 09:53:04,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [5e506f3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ea17f69, log id: 1e8843a8 >2012-12-17 09:53:04,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:05,027 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:53:05,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34351a >2012-12-17 09:53:05,088 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [5e506f3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:53:05,455 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [66369950] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:53:05,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 50915a43 >2012-12-17 09:53:05,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3ba1ae9c >2012-12-17 09:53:05,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:05,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SpmStopVDSCommand, log id: 3ba1ae9c >2012-12-17 09:53:06,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, ResetIrsVDSCommand, log id: 50915a43 >2012-12-17 09:53:06,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 171eb9a2 >2012-12-17 09:53:06,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:06,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 171eb9a2 >2012-12-17 09:53:07,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:08,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4989 >2012-12-17 09:53:08,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4989, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f2d71aa >2012-12-17 09:53:08,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling started: taskId = f9a714c9-7bd7-4279-8065-9104d521a7d7 >2012-12-17 09:53:09,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling ended: taskId = f9a714c9-7bd7-4279-8065-9104d521a7d7 task status = finished >2012-12-17 09:53:09,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] spmStart polling ended, spm status: SPM >2012-12-17 09:53:09,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9a714c9-7bd7-4279-8065-9104d521a7d7), log id: 4afa653f >2012-12-17 09:53:09,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMClearTaskVDSCommand, log id: 4afa653f >2012-12-17 09:53:09,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d0d318d, log id: 2f2d71aa >2012-12-17 09:53:09,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:09,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:09,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35fc7a7d >2012-12-17 09:53:09,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35fc7a7d >2012-12-17 09:53:09,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34351a >2012-12-17 09:53:09,442 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:53:09,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2547293c >2012-12-17 09:53:09,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:09,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4846def2 >2012-12-17 09:53:09,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4846def2 >2012-12-17 09:53:09,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-64) [2353b939] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2547293c >2012-12-17 09:53:09,462 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) [2353b939] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:53:18,046 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [681f9614] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:53:18,470 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [3c62925a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:53:18,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ab9ce52 >2012-12-17 09:53:18,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6b203c3c >2012-12-17 09:53:18,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:18,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, SpmStopVDSCommand, log id: 6b203c3c >2012-12-17 09:53:18,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, ResetIrsVDSCommand, log id: ab9ce52 >2012-12-17 09:53:18,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29e46064 >2012-12-17 09:53:18,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:18,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29e46064 >2012-12-17 09:53:19,432 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:53:29,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:29,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4990 >2012-12-17 09:53:29,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4990, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79c9d4a2 >2012-12-17 09:53:29,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling started: taskId = f52a7acc-63b5-40b4-8d9e-fd01a6c452b7 >2012-12-17 09:53:30,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended: taskId = f52a7acc-63b5-40b4-8d9e-fd01a6c452b7 task status = finished >2012-12-17 09:53:30,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] spmStart polling ended, spm status: SPM >2012-12-17 09:53:30,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f52a7acc-63b5-40b4-8d9e-fd01a6c452b7), log id: 5dea7eb9 >2012-12-17 09:53:30,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, HSMClearTaskVDSCommand, log id: 5dea7eb9 >2012-12-17 09:53:30,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [56a7c4a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4e25d3da, log id: 79c9d4a2 >2012-12-17 09:53:30,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:30,475 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:53:30,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29fff22e >2012-12-17 09:53:30,524 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [56a7c4a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:53:31,996 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [1ecb2997] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:53:32,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79c7dbf9 >2012-12-17 09:53:32,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 53f6222b >2012-12-17 09:53:32,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:32,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, SpmStopVDSCommand, log id: 53f6222b >2012-12-17 09:53:32,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, ResetIrsVDSCommand, log id: 79c7dbf9 >2012-12-17 09:53:32,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ead5ad6 >2012-12-17 09:53:32,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:32,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1ecb2997] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ead5ad6 >2012-12-17 09:53:32,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:32,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4991 >2012-12-17 09:53:32,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4991, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bf58e07 >2012-12-17 09:53:32,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling started: taskId = 042d28b5-70a2-4749-a93a-781366de8a52 >2012-12-17 09:53:33,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended: taskId = 042d28b5-70a2-4749-a93a-781366de8a52 task status = finished >2012-12-17 09:53:33,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended, spm status: SPM >2012-12-17 09:53:33,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=042d28b5-70a2-4749-a93a-781366de8a52), log id: 44e898c0 >2012-12-17 09:53:33,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMClearTaskVDSCommand, log id: 44e898c0 >2012-12-17 09:53:33,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2848898, log id: 3bf58e07 >2012-12-17 09:53:33,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:33,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:33,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2528e505 >2012-12-17 09:53:33,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2528e505 >2012-12-17 09:53:33,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 29fff22e >2012-12-17 09:53:33,818 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:53:33,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 67409d1 >2012-12-17 09:53:33,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:33,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 73535fe5 >2012-12-17 09:53:33,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 73535fe5 >2012-12-17 09:53:33,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 67409d1 >2012-12-17 09:53:33,838 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-94) [2b017a7d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:53:40,426 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:53:42,689 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [54d6e7f6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:53:42,987 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-53) [5bc6d47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:53:43,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a20f1b9 >2012-12-17 09:53:43,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 78a821b2 >2012-12-17 09:53:43,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:43,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, SpmStopVDSCommand, log id: 78a821b2 >2012-12-17 09:53:43,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, ResetIrsVDSCommand, log id: 7a20f1b9 >2012-12-17 09:53:43,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 476b2022 >2012-12-17 09:53:43,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [5bc6d47] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:43,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-53) [5bc6d47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 476b2022 >2012-12-17 09:53:51,493 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:53:53,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:53,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4992 >2012-12-17 09:53:53,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4992, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 771738d9 >2012-12-17 09:53:53,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling started: taskId = 93d6eb0f-e76f-4f0c-970f-81e1a6ae4981 >2012-12-17 09:53:54,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended: taskId = 93d6eb0f-e76f-4f0c-970f-81e1a6ae4981 task status = finished >2012-12-17 09:53:54,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] spmStart polling ended, spm status: SPM >2012-12-17 09:53:54,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93d6eb0f-e76f-4f0c-970f-81e1a6ae4981), log id: 2b645912 >2012-12-17 09:53:54,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, HSMClearTaskVDSCommand, log id: 2b645912 >2012-12-17 09:53:54,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [36643dd7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4913c2ff, log id: 771738d9 >2012-12-17 09:53:54,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:54,838 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:53:54,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f91b12e >2012-12-17 09:53:54,899 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [36643dd7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:53:55,382 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [37e97c0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:53:55,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1bdbd6e8 >2012-12-17 09:53:55,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 684b0a2e >2012-12-17 09:53:55,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:55,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, SpmStopVDSCommand, log id: 684b0a2e >2012-12-17 09:53:55,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, ResetIrsVDSCommand, log id: 1bdbd6e8 >2012-12-17 09:53:55,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e65c77e >2012-12-17 09:53:55,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:55,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e65c77e >2012-12-17 09:53:56,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:53:56,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4993 >2012-12-17 09:53:56,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4993, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ed3bf3a >2012-12-17 09:53:56,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling started: taskId = 98e723ed-198b-4282-8030-d333e5ca3700 >2012-12-17 09:53:58,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling ended: taskId = 98e723ed-198b-4282-8030-d333e5ca3700 task status = finished >2012-12-17 09:53:58,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] spmStart polling ended, spm status: SPM >2012-12-17 09:53:58,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=98e723ed-198b-4282-8030-d333e5ca3700), log id: 5297a880 >2012-12-17 09:53:58,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMClearTaskVDSCommand, log id: 5297a880 >2012-12-17 09:53:58,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62b795dc, log id: 2ed3bf3a >2012-12-17 09:53:58,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:53:58,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:58,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78b94f3f >2012-12-17 09:53:59,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78b94f3f >2012-12-17 09:53:59,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6f91b12e >2012-12-17 09:53:59,011 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:53:59,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4e01ef04 >2012-12-17 09:53:59,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:53:59,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3238bcd5 >2012-12-17 09:53:59,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3238bcd5 >2012-12-17 09:53:59,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-79) [37d03d47] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e01ef04 >2012-12-17 09:53:59,031 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) [37d03d47] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:54:02,239 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:54:07,689 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [469720a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:54:08,128 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [384b399c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:54:08,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 40a4f8b6 >2012-12-17 09:54:08,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4718d6d1 >2012-12-17 09:54:08,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:08,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStopVDSCommand, log id: 4718d6d1 >2012-12-17 09:54:08,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, ResetIrsVDSCommand, log id: 40a4f8b6 >2012-12-17 09:54:08,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50229f69 >2012-12-17 09:54:08,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:08,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50229f69 >2012-12-17 09:54:12,443 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:54:18,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:54:19,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4994 >2012-12-17 09:54:19,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4994, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e67cd6 >2012-12-17 09:54:19,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling started: taskId = 8061c79b-2442-4321-9bbb-c6e2d8c35ea2 >2012-12-17 09:54:20,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling ended: taskId = 8061c79b-2442-4321-9bbb-c6e2d8c35ea2 task status = finished >2012-12-17 09:54:20,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] spmStart polling ended, spm status: SPM >2012-12-17 09:54:20,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [79357c7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8061c79b-2442-4321-9bbb-c6e2d8c35ea2), log id: 2dc865ac >2012-12-17 09:54:20,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, HSMClearTaskVDSCommand, log id: 2dc865ac >2012-12-17 09:54:20,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [79357c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f8b16af, log id: 15e67cd6 >2012-12-17 09:54:23,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:54:24,776 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:54:24,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2660eea7 >2012-12-17 09:54:24,777 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [79357c7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:54:27,760 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [5f725bc5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:54:27,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2ea5d03c >2012-12-17 09:54:27,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 207c8bee >2012-12-17 09:54:27,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:28,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SpmStopVDSCommand, log id: 207c8bee >2012-12-17 09:54:29,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, ResetIrsVDSCommand, log id: 2ea5d03c >2012-12-17 09:54:29,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a8a85ec >2012-12-17 09:54:29,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:29,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a8a85ec >2012-12-17 09:54:30,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:54:30,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4995 >2012-12-17 09:54:30,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4995, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79614e72 >2012-12-17 09:54:30,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling started: taskId = 2bb5623f-45b5-4701-8187-cd072c0d24a1 >2012-12-17 09:54:31,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling ended: taskId = 2bb5623f-45b5-4701-8187-cd072c0d24a1 task status = finished >2012-12-17 09:54:31,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] spmStart polling ended, spm status: SPM >2012-12-17 09:54:31,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2bb5623f-45b5-4701-8187-cd072c0d24a1), log id: 68c58fa2 >2012-12-17 09:54:31,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMClearTaskVDSCommand, log id: 68c58fa2 >2012-12-17 09:54:31,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2590a6b, log id: 79614e72 >2012-12-17 09:54:32,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:54:32,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:32,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a6efb25 >2012-12-17 09:54:32,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a6efb25 >2012-12-17 09:54:32,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2660eea7 >2012-12-17 09:54:32,484 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:54:32,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c2a6050 >2012-12-17 09:54:32,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:32,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 262bdc64 >2012-12-17 09:54:32,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 262bdc64 >2012-12-17 09:54:32,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-62) [4f3078ed] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c2a6050 >2012-12-17 09:54:32,505 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-62) [4f3078ed] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:54:34,827 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:54:40,370 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [2bcb8475] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:54:40,509 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:54:40,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 408c0baa >2012-12-17 09:54:40,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1b75f841 >2012-12-17 09:54:40,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:40,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStopVDSCommand, log id: 1b75f841 >2012-12-17 09:54:41,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, ResetIrsVDSCommand, log id: 408c0baa >2012-12-17 09:54:41,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d75b817 >2012-12-17 09:54:41,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:41,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d75b817 >2012-12-17 09:54:45,603 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:54:51,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:54:52,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4996 >2012-12-17 09:54:52,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4996, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65f71b9b >2012-12-17 09:54:52,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling started: taskId = 172bd843-5cfb-49c1-957b-962c121bbbf2 >2012-12-17 09:54:53,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended: taskId = 172bd843-5cfb-49c1-957b-962c121bbbf2 task status = finished >2012-12-17 09:54:53,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] spmStart polling ended, spm status: SPM >2012-12-17 09:54:53,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=172bd843-5cfb-49c1-957b-962c121bbbf2), log id: 15ab979e >2012-12-17 09:54:53,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, HSMClearTaskVDSCommand, log id: 15ab979e >2012-12-17 09:54:53,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [3e7d3817] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@346f112f, log id: 65f71b9b >2012-12-17 09:54:53,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:54:53,275 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:54:53,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5588c968 >2012-12-17 09:54:53,294 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [3e7d3817] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:54:53,387 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [40e7f581] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:54:53,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2c7d0548 >2012-12-17 09:54:53,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41992b4e >2012-12-17 09:54:53,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:53,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SpmStopVDSCommand, log id: 41992b4e >2012-12-17 09:54:53,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, ResetIrsVDSCommand, log id: 2c7d0548 >2012-12-17 09:54:53,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 208685f >2012-12-17 09:54:53,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:53,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 208685f >2012-12-17 09:54:53,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:54:53,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4997 >2012-12-17 09:54:53,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4997, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76e1b6dc >2012-12-17 09:54:53,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling started: taskId = e9690ad4-b5f1-4dbf-af4f-7ca004f55df8 >2012-12-17 09:54:54,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling ended: taskId = e9690ad4-b5f1-4dbf-af4f-7ca004f55df8 task status = finished >2012-12-17 09:54:54,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] spmStart polling ended, spm status: SPM >2012-12-17 09:54:54,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e9690ad4-b5f1-4dbf-af4f-7ca004f55df8), log id: 49acaa46 >2012-12-17 09:54:54,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMClearTaskVDSCommand, log id: 49acaa46 >2012-12-17 09:54:54,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66deff73, log id: 76e1b6dc >2012-12-17 09:54:54,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:54:54,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:54,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 721fbf43 >2012-12-17 09:54:54,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 721fbf43 >2012-12-17 09:54:54,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5588c968 >2012-12-17 09:54:54,963 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:54:54,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 423d359c >2012-12-17 09:54:54,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:54:54,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3fa1a9b0 >2012-12-17 09:54:54,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3fa1a9b0 >2012-12-17 09:54:54,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-95) [17eab1eb] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 423d359c >2012-12-17 09:54:54,985 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-95) [17eab1eb] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:54:57,442 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Checking autorecoverable hosts >2012-12-17 09:55:00,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Autorecovering 0 hosts >2012-12-17 09:55:00,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Checking autorecoverable hosts done >2012-12-17 09:55:00,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [384b399c] Autorecovering storage domains is disabled, skipping >2012-12-17 09:55:03,745 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [12d72057] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:55:04,104 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [4007c40f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:55:04,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: ff88fa6 >2012-12-17 09:55:04,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2de6d00f >2012-12-17 09:55:04,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:04,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStopVDSCommand, log id: 2de6d00f >2012-12-17 09:55:04,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, ResetIrsVDSCommand, log id: ff88fa6 >2012-12-17 09:55:04,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 620c33d4 >2012-12-17 09:55:04,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:04,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 620c33d4 >2012-12-17 09:55:08,327 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:55:14,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:55:14,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4998 >2012-12-17 09:55:14,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4998, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66e777e7 >2012-12-17 09:55:14,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling started: taskId = 090598ad-5671-4a10-bb25-103d2aec8b03 >2012-12-17 09:55:15,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended: taskId = 090598ad-5671-4a10-bb25-103d2aec8b03 task status = finished >2012-12-17 09:55:15,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] spmStart polling ended, spm status: SPM >2012-12-17 09:55:15,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=090598ad-5671-4a10-bb25-103d2aec8b03), log id: 1331cc12 >2012-12-17 09:55:15,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, HSMClearTaskVDSCommand, log id: 1331cc12 >2012-12-17 09:55:15,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2b017a7d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a4cbb96, log id: 66e777e7 >2012-12-17 09:55:16,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:55:16,945 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:55:16,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 683d40f >2012-12-17 09:55:16,945 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2b017a7d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:55:17,155 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2669bad8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:55:17,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 664493e8 >2012-12-17 09:55:17,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 395b979e >2012-12-17 09:55:17,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:17,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, SpmStopVDSCommand, log id: 395b979e >2012-12-17 09:55:17,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, ResetIrsVDSCommand, log id: 664493e8 >2012-12-17 09:55:17,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cbc3fbb >2012-12-17 09:55:17,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:17,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cbc3fbb >2012-12-17 09:55:17,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:55:17,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 4999 >2012-12-17 09:55:17,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=4999, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42a51a13 >2012-12-17 09:55:17,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling started: taskId = 37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0 >2012-12-17 09:55:18,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended: taskId = 37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0 task status = finished >2012-12-17 09:55:18,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] spmStart polling ended, spm status: SPM >2012-12-17 09:55:18,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=37b1b2c8-d97f-4cd6-9166-2ec5b5fd29b0), log id: 17275545 >2012-12-17 09:55:18,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMClearTaskVDSCommand, log id: 17275545 >2012-12-17 09:55:18,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@174f30e3, log id: 42a51a13 >2012-12-17 09:55:19,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:55:19,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:19,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20586737 >2012-12-17 09:55:19,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20586737 >2012-12-17 09:55:19,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 683d40f >2012-12-17 09:55:19,293 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:19,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7979e823 >2012-12-17 09:55:19,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:19,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 16565578 >2012-12-17 09:55:19,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16565578 >2012-12-17 09:55:19,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-77) [32133550] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7979e823 >2012-12-17 09:55:19,315 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-77) [32133550] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:19,347 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:55:27,889 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1cbe9fb7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:55:28,236 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [7fb5660a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:55:28,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2d5fdadc >2012-12-17 09:55:28,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4e59b9cc >2012-12-17 09:55:28,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:28,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, SpmStopVDSCommand, log id: 4e59b9cc >2012-12-17 09:55:28,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, ResetIrsVDSCommand, log id: 2d5fdadc >2012-12-17 09:55:28,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46d38d19 >2012-12-17 09:55:28,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [7fb5660a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:28,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [7fb5660a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46d38d19 >2012-12-17 09:55:30,524 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:55:38,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:55:38,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5000 >2012-12-17 09:55:38,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5000, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62454aa3 >2012-12-17 09:55:38,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling started: taskId = ae1b3009-e724-4ba1-9b47-7545b0b31bc8 >2012-12-17 09:55:39,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling ended: taskId = ae1b3009-e724-4ba1-9b47-7545b0b31bc8 task status = finished >2012-12-17 09:55:39,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] spmStart polling ended, spm status: SPM >2012-12-17 09:55:39,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae1b3009-e724-4ba1-9b47-7545b0b31bc8), log id: 6eab61de >2012-12-17 09:55:39,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, HSMClearTaskVDSCommand, log id: 6eab61de >2012-12-17 09:55:39,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [63fee8f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35a66e95, log id: 62454aa3 >2012-12-17 09:55:39,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:55:39,978 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:55:39,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42d67975 >2012-12-17 09:55:40,025 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [63fee8f1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:55:40,390 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [5a33a5b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:55:40,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17dbb04b >2012-12-17 09:55:40,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 12bcee3c >2012-12-17 09:55:40,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:41,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, SpmStopVDSCommand, log id: 12bcee3c >2012-12-17 09:55:41,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, ResetIrsVDSCommand, log id: 17dbb04b >2012-12-17 09:55:41,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c0497ff >2012-12-17 09:55:41,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [5a33a5b6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:41,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [5a33a5b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c0497ff >2012-12-17 09:55:42,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:55:42,252 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:55:42,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5001 >2012-12-17 09:55:42,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5001, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b146a79 >2012-12-17 09:55:42,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = e91ace84-5987-4aa3-8606-efdc49d60b63 >2012-12-17 09:55:43,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = e91ace84-5987-4aa3-8606-efdc49d60b63 task status = finished >2012-12-17 09:55:43,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM >2012-12-17 09:55:43,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e91ace84-5987-4aa3-8606-efdc49d60b63), log id: 167c1698 >2012-12-17 09:55:43,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 167c1698 >2012-12-17 09:55:43,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2caf60b0, log id: 4b146a79 >2012-12-17 09:55:43,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:55:43,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:43,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1a6cc3ed >2012-12-17 09:55:43,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1a6cc3ed >2012-12-17 09:55:43,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 42d67975 >2012-12-17 09:55:43,519 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:43,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5ab30a28 >2012-12-17 09:55:43,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:43,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35204e4 >2012-12-17 09:55:43,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35204e4 >2012-12-17 09:55:43,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5ab30a28 >2012-12-17 09:55:43,539 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:43,551 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:55:43,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: c5a11e8 >2012-12-17 09:55:43,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:43,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 27c1ea80 >2012-12-17 09:55:43,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27c1ea80 >2012-12-17 09:55:43,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c5a11e8 >2012-12-17 09:55:43,572 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:43,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c0a8de4 >2012-12-17 09:55:43,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:43,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 320facef >2012-12-17 09:55:43,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 320facef >2012-12-17 09:55:43,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c0a8de4 >2012-12-17 09:55:43,592 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-25) [40e7f581] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:55:52,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [61b60bdd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:55:53,463 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:55:54,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3644bf4d >2012-12-17 09:55:54,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46929707 >2012-12-17 09:55:54,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:54,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, SpmStopVDSCommand, log id: 46929707 >2012-12-17 09:55:54,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, ResetIrsVDSCommand, log id: 3644bf4d >2012-12-17 09:55:54,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2cc77782 >2012-12-17 09:55:54,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:55:54,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6c52cc2e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2cc77782 >2012-12-17 09:56:03,027 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:56:04,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:56:04,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5002 >2012-12-17 09:56:04,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5002, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64306403 >2012-12-17 09:56:04,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling started: taskId = 58077d2e-1274-420d-97fb-9c6b1a910c8b >2012-12-17 09:56:05,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended: taskId = 58077d2e-1274-420d-97fb-9c6b1a910c8b task status = finished >2012-12-17 09:56:05,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] spmStart polling ended, spm status: SPM >2012-12-17 09:56:05,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=58077d2e-1274-420d-97fb-9c6b1a910c8b), log id: 92f90c8 >2012-12-17 09:56:05,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, HSMClearTaskVDSCommand, log id: 92f90c8 >2012-12-17 09:56:05,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [7a6ed577] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11ca3f2e, log id: 64306403 >2012-12-17 09:56:06,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:56:07,128 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:56:07,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 76cdbd8d >2012-12-17 09:56:07,192 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [7a6ed577] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:56:07,653 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [3ad075f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:56:07,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6aaa3971 >2012-12-17 09:56:07,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 15e80d24 >2012-12-17 09:56:08,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:08,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SpmStopVDSCommand, log id: 15e80d24 >2012-12-17 09:56:08,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, ResetIrsVDSCommand, log id: 6aaa3971 >2012-12-17 09:56:08,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2801f20 >2012-12-17 09:56:08,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:08,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2801f20 >2012-12-17 09:56:08,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:56:08,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5003 >2012-12-17 09:56:08,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5003, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 503dc5a6 >2012-12-17 09:56:08,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling started: taskId = 8a409f6a-44d8-4049-8cf0-bdc80b7be207 >2012-12-17 09:56:09,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended: taskId = 8a409f6a-44d8-4049-8cf0-bdc80b7be207 task status = finished >2012-12-17 09:56:09,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended, spm status: SPM >2012-12-17 09:56:09,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8a409f6a-44d8-4049-8cf0-bdc80b7be207), log id: 22773145 >2012-12-17 09:56:09,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMClearTaskVDSCommand, log id: 22773145 >2012-12-17 09:56:09,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f43bab0, log id: 503dc5a6 >2012-12-17 09:56:09,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:56:09,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:09,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24b455d3 >2012-12-17 09:56:09,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24b455d3 >2012-12-17 09:56:09,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 76cdbd8d >2012-12-17 09:56:09,496 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:56:09,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b6ea278 >2012-12-17 09:56:09,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:09,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 209c800f >2012-12-17 09:56:09,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 209c800f >2012-12-17 09:56:09,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b6ea278 >2012-12-17 09:56:09,517 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [44cd9139] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:56:13,326 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:56:18,399 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [7fb5660a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:56:18,686 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [3577b816] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:56:18,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7d74c9c2 >2012-12-17 09:56:18,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 511179b >2012-12-17 09:56:18,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:18,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, SpmStopVDSCommand, log id: 511179b >2012-12-17 09:56:18,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, ResetIrsVDSCommand, log id: 7d74c9c2 >2012-12-17 09:56:18,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d74fc06 >2012-12-17 09:56:18,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:18,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d74fc06 >2012-12-17 09:56:23,615 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:56:29,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:56:32,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5004 >2012-12-17 09:56:32,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5004, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74a4c31a >2012-12-17 09:56:32,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling started: taskId = 2d7b75f1-caf6-4dae-a63d-000418833f97 >2012-12-17 09:56:33,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended: taskId = 2d7b75f1-caf6-4dae-a63d-000418833f97 task status = finished >2012-12-17 09:56:33,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] spmStart polling ended, spm status: SPM >2012-12-17 09:56:33,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2d7b75f1-caf6-4dae-a63d-000418833f97), log id: 4eac3ff8 >2012-12-17 09:56:33,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, HSMClearTaskVDSCommand, log id: 4eac3ff8 >2012-12-17 09:56:33,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [42560f9e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@56150e49, log id: 74a4c31a >2012-12-17 09:56:33,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:56:34,019 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:56:34,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6cb1fc7e >2012-12-17 09:56:34,027 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [42560f9e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:56:34,201 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [56592212] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:56:34,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a7e6228 >2012-12-17 09:56:34,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 890ab47 >2012-12-17 09:56:34,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:34,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SpmStopVDSCommand, log id: 890ab47 >2012-12-17 09:56:34,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, ResetIrsVDSCommand, log id: 2a7e6228 >2012-12-17 09:56:34,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77eeb7d4 >2012-12-17 09:56:34,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:34,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77eeb7d4 >2012-12-17 09:56:34,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:56:34,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5005 >2012-12-17 09:56:34,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5005, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7a68a922 >2012-12-17 09:56:34,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling started: taskId = dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6 >2012-12-17 09:56:35,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended: taskId = dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6 task status = finished >2012-12-17 09:56:36,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended, spm status: SPM >2012-12-17 09:56:36,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dba95c9f-f7ce-4d7f-85cc-fcbc2e9d13d6), log id: 217bd1c5 >2012-12-17 09:56:36,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMClearTaskVDSCommand, log id: 217bd1c5 >2012-12-17 09:56:36,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4808edaf, log id: 7a68a922 >2012-12-17 09:56:37,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:56:37,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:37,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 445fea70 >2012-12-17 09:56:37,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 445fea70 >2012-12-17 09:56:37,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6cb1fc7e >2012-12-17 09:56:37,345 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:56:37,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 75188221 >2012-12-17 09:56:37,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:37,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c5f658a >2012-12-17 09:56:37,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c5f658a >2012-12-17 09:56:37,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75188221 >2012-12-17 09:56:37,365 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [312ad09c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:56:44,141 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:56:45,063 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [510758b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:56:46,504 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [3e0f1a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:56:47,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20734fd7 >2012-12-17 09:56:47,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 76386c4d >2012-12-17 09:56:47,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:48,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, SpmStopVDSCommand, log id: 76386c4d >2012-12-17 09:56:48,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, ResetIrsVDSCommand, log id: 20734fd7 >2012-12-17 09:56:48,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d5fa654 >2012-12-17 09:56:48,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [3e0f1a8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:56:48,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [3e0f1a8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d5fa654 >2012-12-17 09:56:54,638 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:56:58,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:56:59,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5006 >2012-12-17 09:56:59,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5006, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ede79be >2012-12-17 09:56:59,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling started: taskId = c2f41f76-d2f6-42f8-be61-5e7683a9b45f >2012-12-17 09:57:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended: taskId = c2f41f76-d2f6-42f8-be61-5e7683a9b45f task status = finished >2012-12-17 09:57:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] spmStart polling ended, spm status: SPM >2012-12-17 09:57:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c2f41f76-d2f6-42f8-be61-5e7683a9b45f), log id: 7837eb1c >2012-12-17 09:57:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, HSMClearTaskVDSCommand, log id: 7837eb1c >2012-12-17 09:57:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [44cd9139] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f2cb8ed, log id: 1ede79be >2012-12-17 09:57:00,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:00,376 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:57:00,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 45507629 >2012-12-17 09:57:00,459 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [44cd9139] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:57:00,688 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [536ede7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:57:00,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e86e124 >2012-12-17 09:57:00,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 79a16691 >2012-12-17 09:57:00,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:00,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStopVDSCommand, log id: 79a16691 >2012-12-17 09:57:01,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, ResetIrsVDSCommand, log id: 3e86e124 >2012-12-17 09:57:01,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ac39947 >2012-12-17 09:57:01,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:01,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ac39947 >2012-12-17 09:57:04,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:57:04,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5007 >2012-12-17 09:57:04,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5007, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7daa7153 >2012-12-17 09:57:04,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling started: taskId = 0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991 >2012-12-17 09:57:05,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended: taskId = 0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991 task status = finished >2012-12-17 09:57:05,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended, spm status: SPM >2012-12-17 09:57:05,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0fd174e1-2ea2-43a5-a2c0-e8f0ab1fe991), log id: 2c386a32 >2012-12-17 09:57:05,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMClearTaskVDSCommand, log id: 2c386a32 >2012-12-17 09:57:05,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bc40f4c, log id: 7daa7153 >2012-12-17 09:57:05,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:05,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:05,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 736e357e >2012-12-17 09:57:05,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 736e357e >2012-12-17 09:57:05,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 45507629 >2012-12-17 09:57:05,799 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:05,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 487a1e66 >2012-12-17 09:57:05,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:05,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: b8d33f1 >2012-12-17 09:57:05,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: b8d33f1 >2012-12-17 09:57:05,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 487a1e66 >2012-12-17 09:57:05,819 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-32) [3e0f3441] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:14,251 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [53e9685e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:57:15,151 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [415bb057] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:57:15,474 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:57:15,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57da980e >2012-12-17 09:57:15,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3e25539e >2012-12-17 09:57:15,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:15,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, SpmStopVDSCommand, log id: 3e25539e >2012-12-17 09:57:16,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, ResetIrsVDSCommand, log id: 57da980e >2012-12-17 09:57:17,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [415bb057] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25a8f3d4 >2012-12-17 09:57:17,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [415bb057] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:17,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [415bb057] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25a8f3d4 >2012-12-17 09:57:27,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:57:27,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5008 >2012-12-17 09:57:27,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5008, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37629d2a >2012-12-17 09:57:27,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling started: taskId = 95ff63ea-9899-4692-87a0-69e530cfe11e >2012-12-17 09:57:28,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended: taskId = 95ff63ea-9899-4692-87a0-69e530cfe11e task status = finished >2012-12-17 09:57:28,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] spmStart polling ended, spm status: SPM >2012-12-17 09:57:28,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=95ff63ea-9899-4692-87a0-69e530cfe11e), log id: 39d24fa5 >2012-12-17 09:57:28,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, HSMClearTaskVDSCommand, log id: 39d24fa5 >2012-12-17 09:57:28,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [312ad09c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e6af74b, log id: 37629d2a >2012-12-17 09:57:29,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:29,179 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:57:29,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a16a880 >2012-12-17 09:57:29,215 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [312ad09c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:57:29,791 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [140420a5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:57:29,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14c17f35 >2012-12-17 09:57:29,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4947f3c7 >2012-12-17 09:57:29,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:30,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SpmStopVDSCommand, log id: 4947f3c7 >2012-12-17 09:57:30,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, ResetIrsVDSCommand, log id: 14c17f35 >2012-12-17 09:57:30,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76798dc >2012-12-17 09:57:30,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:30,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76798dc >2012-12-17 09:57:30,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:57:31,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5009 >2012-12-17 09:57:31,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5009, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e5737be >2012-12-17 09:57:31,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling started: taskId = e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d >2012-12-17 09:57:32,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended: taskId = e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d task status = finished >2012-12-17 09:57:32,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended, spm status: SPM >2012-12-17 09:57:32,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e09e49d7-9f36-4aa9-bdb8-fad6d11a0b8d), log id: 7fb90016 >2012-12-17 09:57:32,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMClearTaskVDSCommand, log id: 7fb90016 >2012-12-17 09:57:32,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ab4e5dc, log id: 3e5737be >2012-12-17 09:57:32,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:32,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:32,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 114c67d6 >2012-12-17 09:57:32,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 114c67d6 >2012-12-17 09:57:32,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a16a880 >2012-12-17 09:57:32,910 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:32,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f351b79 >2012-12-17 09:57:32,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:32,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60ea5b8 >2012-12-17 09:57:32,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60ea5b8 >2012-12-17 09:57:32,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2f351b79 >2012-12-17 09:57:32,929 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:38,157 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:57:41,955 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4f15776f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:57:42,325 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:57:42,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7beff2fe >2012-12-17 09:57:42,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d7942ad >2012-12-17 09:57:42,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:42,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, SpmStopVDSCommand, log id: 3d7942ad >2012-12-17 09:57:42,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, ResetIrsVDSCommand, log id: 7beff2fe >2012-12-17 09:57:42,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f6bba3b >2012-12-17 09:57:42,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:42,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f6bba3b >2012-12-17 09:57:49,289 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:57:53,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:57:53,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5010 >2012-12-17 09:57:53,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5010, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 37dc1dab >2012-12-17 09:57:53,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling started: taskId = e475a84d-4da0-4de9-a8fc-f4fda9a644a8 >2012-12-17 09:57:54,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling ended: taskId = e475a84d-4da0-4de9-a8fc-f4fda9a644a8 task status = finished >2012-12-17 09:57:54,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] spmStart polling ended, spm status: SPM >2012-12-17 09:57:54,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e475a84d-4da0-4de9-a8fc-f4fda9a644a8), log id: 182d66ef >2012-12-17 09:57:54,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, HSMClearTaskVDSCommand, log id: 182d66ef >2012-12-17 09:57:54,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [5f4cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4cc0e14f, log id: 37dc1dab >2012-12-17 09:57:54,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:54,496 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:57:54,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5fdbb076 >2012-12-17 09:57:54,570 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [5f4cecb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:57:54,916 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [604cecb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:57:54,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 490684b9 >2012-12-17 09:57:54,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 27090b0 >2012-12-17 09:57:54,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:55,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStopVDSCommand, log id: 27090b0 >2012-12-17 09:57:55,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, ResetIrsVDSCommand, log id: 490684b9 >2012-12-17 09:57:55,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 377d20fe >2012-12-17 09:57:55,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:55,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 377d20fe >2012-12-17 09:57:55,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:57:55,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5011 >2012-12-17 09:57:55,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5011, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6245bf90 >2012-12-17 09:57:55,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling started: taskId = 93be4e3f-0abc-4060-80f7-fba0139f4181 >2012-12-17 09:57:56,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended: taskId = 93be4e3f-0abc-4060-80f7-fba0139f4181 task status = finished >2012-12-17 09:57:56,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended, spm status: SPM >2012-12-17 09:57:56,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93be4e3f-0abc-4060-80f7-fba0139f4181), log id: 3592bd3c >2012-12-17 09:57:56,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMClearTaskVDSCommand, log id: 3592bd3c >2012-12-17 09:57:56,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34c5c99e, log id: 6245bf90 >2012-12-17 09:57:57,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:57:57,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:57,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2f1ad9be >2012-12-17 09:57:57,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f1ad9be >2012-12-17 09:57:57,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5fdbb076 >2012-12-17 09:57:57,275 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:57,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 655a6763 >2012-12-17 09:57:57,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:57:57,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 12e476f >2012-12-17 09:57:57,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 12e476f >2012-12-17 09:57:57,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 655a6763 >2012-12-17 09:57:57,295 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:57:59,770 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:58:05,326 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [95f0e17] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:58:05,773 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [6a7edb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:58:07,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 79b4d24e >2012-12-17 09:58:07,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 20d4cb71 >2012-12-17 09:58:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:07,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, SpmStopVDSCommand, log id: 20d4cb71 >2012-12-17 09:58:07,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, ResetIrsVDSCommand, log id: 79b4d24e >2012-12-17 09:58:07,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e310a2a >2012-12-17 09:58:07,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6a7edb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:07,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [6a7edb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e310a2a >2012-12-17 09:58:10,114 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:58:17,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:58:18,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5012 >2012-12-17 09:58:18,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5012, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e8c9236 >2012-12-17 09:58:18,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling started: taskId = ae5cdf62-8231-44be-8d23-de147d9e6df2 >2012-12-17 09:58:19,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended: taskId = ae5cdf62-8231-44be-8d23-de147d9e6df2 task status = finished >2012-12-17 09:58:19,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] spmStart polling ended, spm status: SPM >2012-12-17 09:58:19,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ae5cdf62-8231-44be-8d23-de147d9e6df2), log id: 6d2257e9 >2012-12-17 09:58:19,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, HSMClearTaskVDSCommand, log id: 6d2257e9 >2012-12-17 09:58:19,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [e893ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69b9b9f8, log id: 5e8c9236 >2012-12-17 09:58:19,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:58:19,600 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:58:19,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d2150de >2012-12-17 09:58:19,682 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [e893ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:58:20,158 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [535051e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:58:20,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 595b8fa >2012-12-17 09:58:20,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: b927457 >2012-12-17 09:58:20,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:20,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStopVDSCommand, log id: b927457 >2012-12-17 09:58:20,476 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:58:20,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, ResetIrsVDSCommand, log id: 595b8fa >2012-12-17 09:58:20,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a10b439 >2012-12-17 09:58:20,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:20,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a10b439 >2012-12-17 09:58:21,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:58:22,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5013 >2012-12-17 09:58:22,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5013, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30d597ef >2012-12-17 09:58:22,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = 5ea288a2-b1ed-42b4-8928-adbfd8f90816 >2012-12-17 09:58:23,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = 5ea288a2-b1ed-42b4-8928-adbfd8f90816 task status = finished >2012-12-17 09:58:23,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM >2012-12-17 09:58:23,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5ea288a2-b1ed-42b4-8928-adbfd8f90816), log id: 12e9c602 >2012-12-17 09:58:23,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 12e9c602 >2012-12-17 09:58:23,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31c6b701, log id: 30d597ef >2012-12-17 09:58:23,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:58:23,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:23,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7751b068 >2012-12-17 09:58:23,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7751b068 >2012-12-17 09:58:23,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d2150de >2012-12-17 09:58:23,833 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:23,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1414b90b >2012-12-17 09:58:23,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:23,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 646b8a5d >2012-12-17 09:58:23,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 646b8a5d >2012-12-17 09:58:23,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1414b90b >2012-12-17 09:58:23,853 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:23,904 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:58:23,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d89d6e1 >2012-12-17 09:58:23,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:23,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31ed7e79 >2012-12-17 09:58:23,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31ed7e79 >2012-12-17 09:58:23,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d89d6e1 >2012-12-17 09:58:23,925 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:23,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f0a538a >2012-12-17 09:58:23,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:23,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d402104 >2012-12-17 09:58:23,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d402104 >2012-12-17 09:58:23,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f0a538a >2012-12-17 09:58:23,944 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-47) [7c8941c5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:32,311 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [477daf1e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:58:33,360 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [193f8ef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:58:33,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 710269ea >2012-12-17 09:58:33,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 44e9741 >2012-12-17 09:58:33,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:33,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, SpmStopVDSCommand, log id: 44e9741 >2012-12-17 09:58:33,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, ResetIrsVDSCommand, log id: 710269ea >2012-12-17 09:58:33,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24b94ea1 >2012-12-17 09:58:33,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [193f8ef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:33,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [193f8ef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24b94ea1 >2012-12-17 09:58:42,826 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:58:44,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:58:44,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5014 >2012-12-17 09:58:44,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5014, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e3007a2 >2012-12-17 09:58:44,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling started: taskId = 30af2b92-acd2-4e1a-a327-5fe32a8375d5 >2012-12-17 09:58:45,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling ended: taskId = 30af2b92-acd2-4e1a-a327-5fe32a8375d5 task status = finished >2012-12-17 09:58:45,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] spmStart polling ended, spm status: SPM >2012-12-17 09:58:45,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [537845d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=30af2b92-acd2-4e1a-a327-5fe32a8375d5), log id: 76e1ca5e >2012-12-17 09:58:45,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, HSMClearTaskVDSCommand, log id: 76e1ca5e >2012-12-17 09:58:45,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [537845d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61ad1e4b, log id: 5e3007a2 >2012-12-17 09:58:45,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:58:45,391 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:58:45,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 119ba3e1 >2012-12-17 09:58:45,411 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [537845d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:58:45,602 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [56dc6659] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:58:46,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14b4fcfe >2012-12-17 09:58:46,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 752a5a8 >2012-12-17 09:58:46,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:47,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStopVDSCommand, log id: 752a5a8 >2012-12-17 09:58:47,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, ResetIrsVDSCommand, log id: 14b4fcfe >2012-12-17 09:58:47,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d6c6f75 >2012-12-17 09:58:47,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:47,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d6c6f75 >2012-12-17 09:58:47,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:58:47,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5015 >2012-12-17 09:58:47,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5015, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76287059 >2012-12-17 09:58:47,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling started: taskId = 430c9f1b-25d0-4e9e-a0ff-57d96393cc9e >2012-12-17 09:58:48,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended: taskId = 430c9f1b-25d0-4e9e-a0ff-57d96393cc9e task status = finished >2012-12-17 09:58:48,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended, spm status: SPM >2012-12-17 09:58:48,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=430c9f1b-25d0-4e9e-a0ff-57d96393cc9e), log id: a95929a >2012-12-17 09:58:48,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMClearTaskVDSCommand, log id: a95929a >2012-12-17 09:58:48,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61cbc1e3, log id: 76287059 >2012-12-17 09:58:48,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:58:48,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:48,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3bebaa66 >2012-12-17 09:58:48,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bebaa66 >2012-12-17 09:58:48,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 119ba3e1 >2012-12-17 09:58:48,663 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:48,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d002d45 >2012-12-17 09:58:48,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:48,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22c0020 >2012-12-17 09:58:48,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22c0020 >2012-12-17 09:58:48,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d002d45 >2012-12-17 09:58:48,684 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) [604cecb9] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:58:53,893 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:58:57,566 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [405e01] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:58:57,958 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [406c9a80] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:58:58,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 56c5744f >2012-12-17 09:58:58,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 21f676b7 >2012-12-17 09:58:58,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:58,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, SpmStopVDSCommand, log id: 21f676b7 >2012-12-17 09:58:58,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, ResetIrsVDSCommand, log id: 56c5744f >2012-12-17 09:58:58,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c6377b2 >2012-12-17 09:58:58,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:58:58,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [406c9a80] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c6377b2 >2012-12-17 09:59:04,463 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:59:08,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:59:08,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5016 >2012-12-17 09:59:08,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5016, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22e644e8 >2012-12-17 09:59:08,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling started: taskId = d73944a7-d20d-43ad-9fbc-0b598b4b06f6 >2012-12-17 09:59:09,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling ended: taskId = d73944a7-d20d-43ad-9fbc-0b598b4b06f6 task status = finished >2012-12-17 09:59:09,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] spmStart polling ended, spm status: SPM >2012-12-17 09:59:09,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [2b52624] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d73944a7-d20d-43ad-9fbc-0b598b4b06f6), log id: 49fbbc3d >2012-12-17 09:59:09,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, HSMClearTaskVDSCommand, log id: 49fbbc3d >2012-12-17 09:59:09,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [2b52624] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e036ebd, log id: 22e644e8 >2012-12-17 09:59:10,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:59:10,112 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:59:10,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4dbfc446 >2012-12-17 09:59:10,218 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [2b52624] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:59:10,512 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [8c9ee2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:59:11,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 8920f4e >2012-12-17 09:59:11,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60a765cc >2012-12-17 09:59:11,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:11,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, SpmStopVDSCommand, log id: 60a765cc >2012-12-17 09:59:12,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, ResetIrsVDSCommand, log id: 8920f4e >2012-12-17 09:59:12,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7edca4bb >2012-12-17 09:59:12,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:12,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7edca4bb >2012-12-17 09:59:12,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:59:12,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5017 >2012-12-17 09:59:12,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5017, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5620168f >2012-12-17 09:59:12,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = 10869372-5f03-4651-b0bd-15111239b91c >2012-12-17 09:59:13,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = 10869372-5f03-4651-b0bd-15111239b91c task status = finished >2012-12-17 09:59:13,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM >2012-12-17 09:59:13,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=10869372-5f03-4651-b0bd-15111239b91c), log id: 56447a8e >2012-12-17 09:59:13,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 56447a8e >2012-12-17 09:59:13,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29ca2ce1, log id: 5620168f >2012-12-17 09:59:13,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:59:13,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:13,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 211aace3 >2012-12-17 09:59:13,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 211aace3 >2012-12-17 09:59:13,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4dbfc446 >2012-12-17 09:59:13,626 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:13,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5e4c7c0f >2012-12-17 09:59:13,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:13,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e65799e >2012-12-17 09:59:13,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e65799e >2012-12-17 09:59:13,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e4c7c0f >2012-12-17 09:59:13,646 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:15,030 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:59:22,489 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [3499e3ae] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:59:23,283 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [7d7168a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:59:23,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1287df40 >2012-12-17 09:59:23,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6dd1a0e0 >2012-12-17 09:59:23,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:24,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, SpmStopVDSCommand, log id: 6dd1a0e0 >2012-12-17 09:59:24,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, ResetIrsVDSCommand, log id: 1287df40 >2012-12-17 09:59:24,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4155e423 >2012-12-17 09:59:24,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [7d7168a8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:24,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [7d7168a8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4155e423 >2012-12-17 09:59:27,301 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:59:34,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:59:34,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5018 >2012-12-17 09:59:34,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5018, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e03937d >2012-12-17 09:59:34,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling started: taskId = 1829fd44-17e2-4583-be77-716d4e79443a >2012-12-17 09:59:35,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling ended: taskId = 1829fd44-17e2-4583-be77-716d4e79443a task status = finished >2012-12-17 09:59:35,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] spmStart polling ended, spm status: SPM >2012-12-17 09:59:35,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1829fd44-17e2-4583-be77-716d4e79443a), log id: 95e79e6 >2012-12-17 09:59:35,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, HSMClearTaskVDSCommand, log id: 95e79e6 >2012-12-17 09:59:35,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [63eed75e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1994efd8, log id: e03937d >2012-12-17 09:59:35,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:59:36,797 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:59:36,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 32e7e680 >2012-12-17 09:59:37,057 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [63eed75e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:59:37,419 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [6827474] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:59:37,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 36b39bf6 >2012-12-17 09:59:37,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3d031a1c >2012-12-17 09:59:37,867 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 09:59:37,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:38,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SpmStopVDSCommand, log id: 3d031a1c >2012-12-17 09:59:38,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, ResetIrsVDSCommand, log id: 36b39bf6 >2012-12-17 09:59:38,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17069c68 >2012-12-17 09:59:38,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:38,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17069c68 >2012-12-17 09:59:38,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 09:59:38,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5019 >2012-12-17 09:59:38,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5019, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58d3dc08 >2012-12-17 09:59:38,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling started: taskId = 9c22f530-d793-4b5b-9638-470729d49d8a >2012-12-17 09:59:39,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended: taskId = 9c22f530-d793-4b5b-9638-470729d49d8a task status = finished >2012-12-17 09:59:39,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended, spm status: SPM >2012-12-17 09:59:39,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9c22f530-d793-4b5b-9638-470729d49d8a), log id: 42b5cc43 >2012-12-17 09:59:39,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMClearTaskVDSCommand, log id: 42b5cc43 >2012-12-17 09:59:39,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d7564ec, log id: 58d3dc08 >2012-12-17 09:59:39,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 09:59:39,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:39,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29975f3d >2012-12-17 09:59:40,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29975f3d >2012-12-17 09:59:40,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32e7e680 >2012-12-17 09:59:40,013 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:40,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58c88a25 >2012-12-17 09:59:40,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:40,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e9e0e46 >2012-12-17 09:59:40,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e9e0e46 >2012-12-17 09:59:40,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58c88a25 >2012-12-17 09:59:40,033 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:40,046 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 09:59:40,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6132f83c >2012-12-17 09:59:40,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:40,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4de561b3 >2012-12-17 09:59:40,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4de561b3 >2012-12-17 09:59:40,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6132f83c >2012-12-17 09:59:40,066 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:40,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e7156b9 >2012-12-17 09:59:40,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:40,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13e4db80 >2012-12-17 09:59:40,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13e4db80 >2012-12-17 09:59:40,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e7156b9 >2012-12-17 09:59:40,086 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 09:59:48,836 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [37d03d47] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 09:59:49,372 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [470bd088] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 09:59:49,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 68f47436 >2012-12-17 09:59:49,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 622fde75 >2012-12-17 09:59:49,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:50,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, SpmStopVDSCommand, log id: 622fde75 >2012-12-17 09:59:50,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, ResetIrsVDSCommand, log id: 68f47436 >2012-12-17 09:59:50,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f8aa7ff >2012-12-17 09:59:50,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 09:59:50,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f8aa7ff >2012-12-17 09:59:59,087 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts >2012-12-17 10:00:00,107 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering 0 hosts >2012-12-17 10:00:00,107 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts done >2012-12-17 10:00:00,107 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering storage domains is disabled, skipping >2012-12-17 10:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:00,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5020 >2012-12-17 10:00:00,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5020, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c33d71 >2012-12-17 10:00:00,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling started: taskId = 0a344dba-9a60-4bd2-ad99-0852d7a0f996 >2012-12-17 10:00:01,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling ended: taskId = 0a344dba-9a60-4bd2-ad99-0852d7a0f996 task status = finished >2012-12-17 10:00:01,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] spmStart polling ended, spm status: SPM >2012-12-17 10:00:01,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0a344dba-9a60-4bd2-ad99-0852d7a0f996), log id: 4418f5fa >2012-12-17 10:00:01,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, HSMClearTaskVDSCommand, log id: 4418f5fa >2012-12-17 10:00:01,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [8c9ee2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42bd1303, log id: 6c33d71 >2012-12-17 10:00:01,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:02,181 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:00:02,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ccb5ffb >2012-12-17 10:00:02,239 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [8c9ee2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:00:03,145 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [786860d5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:00:03,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3ad38373 >2012-12-17 10:00:03,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 48c20b0f >2012-12-17 10:00:03,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:03,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SpmStopVDSCommand, log id: 48c20b0f >2012-12-17 10:00:03,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, ResetIrsVDSCommand, log id: 3ad38373 >2012-12-17 10:00:03,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b0e5dc9 >2012-12-17 10:00:03,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:03,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b0e5dc9 >2012-12-17 10:00:03,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:03,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5021 >2012-12-17 10:00:03,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5021, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 398ffdad >2012-12-17 10:00:03,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = f3c01183-d47e-46ac-9212-acce2284526e >2012-12-17 10:00:04,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = f3c01183-d47e-46ac-9212-acce2284526e task status = finished >2012-12-17 10:00:04,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM >2012-12-17 10:00:04,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3c01183-d47e-46ac-9212-acce2284526e), log id: 2893e673 >2012-12-17 10:00:04,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 2893e673 >2012-12-17 10:00:04,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5843abb4, log id: 398ffdad >2012-12-17 10:00:05,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:05,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:05,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 253c5937 >2012-12-17 10:00:05,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 253c5937 >2012-12-17 10:00:05,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ccb5ffb >2012-12-17 10:00:05,273 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:00:05,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1d053cfa >2012-12-17 10:00:05,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:05,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6e3b3a7f >2012-12-17 10:00:05,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e3b3a7f >2012-12-17 10:00:05,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d053cfa >2012-12-17 10:00:05,294 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:00:09,596 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:00:13,861 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [7d7168a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:00:14,072 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [44b78d4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:00:14,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 219606f7 >2012-12-17 10:00:14,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 344864ee >2012-12-17 10:00:14,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:14,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, SpmStopVDSCommand, log id: 344864ee >2012-12-17 10:00:14,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, ResetIrsVDSCommand, log id: 219606f7 >2012-12-17 10:00:14,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b4f1b2b >2012-12-17 10:00:14,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [44b78d4a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:14,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [44b78d4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b4f1b2b >2012-12-17 10:00:19,890 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:00:24,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:24,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5022 >2012-12-17 10:00:24,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5022, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 301bfacd >2012-12-17 10:00:24,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling started: taskId = 93be342a-9207-4c98-835e-8f19c2ed69c6 >2012-12-17 10:00:25,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling ended: taskId = 93be342a-9207-4c98-835e-8f19c2ed69c6 task status = finished >2012-12-17 10:00:25,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] spmStart polling ended, spm status: SPM >2012-12-17 10:00:25,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=93be342a-9207-4c98-835e-8f19c2ed69c6), log id: 48372c43 >2012-12-17 10:00:25,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, HSMClearTaskVDSCommand, log id: 48372c43 >2012-12-17 10:00:25,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [4f008eee] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e3b07bf, log id: 301bfacd >2012-12-17 10:00:25,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:27,261 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:00:27,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 31857b5a >2012-12-17 10:00:27,406 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [4f008eee] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:00:28,211 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [2a110a6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:00:28,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4804d7f5 >2012-12-17 10:00:28,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f3243f8 >2012-12-17 10:00:28,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:28,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, SpmStopVDSCommand, log id: 4f3243f8 >2012-12-17 10:00:28,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, ResetIrsVDSCommand, log id: 4804d7f5 >2012-12-17 10:00:28,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29e5a356 >2012-12-17 10:00:28,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:28,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [2a110a6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29e5a356 >2012-12-17 10:00:28,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:28,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5023 >2012-12-17 10:00:28,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5023, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64bd0a5e >2012-12-17 10:00:29,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = 665e0c31-21f9-4cc3-a366-7235eb915651 >2012-12-17 10:00:30,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = 665e0c31-21f9-4cc3-a366-7235eb915651 task status = finished >2012-12-17 10:00:30,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM >2012-12-17 10:00:30,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=665e0c31-21f9-4cc3-a366-7235eb915651), log id: 47b3c6b3 >2012-12-17 10:00:30,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 47b3c6b3 >2012-12-17 10:00:30,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29bc2cdb, log id: 64bd0a5e >2012-12-17 10:00:30,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:30,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:30,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 52b91cb0 >2012-12-17 10:00:30,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 52b91cb0 >2012-12-17 10:00:30,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 31857b5a >2012-12-17 10:00:30,394 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:00:30,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 414935d8 >2012-12-17 10:00:30,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:30,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4a67838c >2012-12-17 10:00:30,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4a67838c >2012-12-17 10:00:30,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 414935d8 >2012-12-17 10:00:30,414 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:00:31,230 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:00:38,920 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [5a33a5b6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:00:39,078 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [4d407d58] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:00:39,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5d04f55d >2012-12-17 10:00:39,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 655cf38d >2012-12-17 10:00:39,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:39,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, SpmStopVDSCommand, log id: 655cf38d >2012-12-17 10:00:39,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, ResetIrsVDSCommand, log id: 5d04f55d >2012-12-17 10:00:39,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 769099 >2012-12-17 10:00:39,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:39,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 769099 >2012-12-17 10:00:42,777 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:00:49,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:49,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5024 >2012-12-17 10:00:49,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5024, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44471110 >2012-12-17 10:00:49,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling started: taskId = eab553e4-00a3-49d6-957d-c81c8f28adf9 >2012-12-17 10:00:50,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling ended: taskId = eab553e4-00a3-49d6-957d-c81c8f28adf9 task status = finished >2012-12-17 10:00:51,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] spmStart polling ended, spm status: SPM >2012-12-17 10:00:51,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eab553e4-00a3-49d6-957d-c81c8f28adf9), log id: 21682b71 >2012-12-17 10:00:51,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, HSMClearTaskVDSCommand, log id: 21682b71 >2012-12-17 10:00:51,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [33bbf9d8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3dfc18d9, log id: 44471110 >2012-12-17 10:00:52,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:52,948 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:00:52,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1421d86b >2012-12-17 10:00:53,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [33bbf9d8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:00:53,567 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [742a6686] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:00:53,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 76d8fca7 >2012-12-17 10:00:53,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a9b6d0b >2012-12-17 10:00:53,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:53,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, SpmStopVDSCommand, log id: 6a9b6d0b >2012-12-17 10:00:53,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, ResetIrsVDSCommand, log id: 76d8fca7 >2012-12-17 10:00:53,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [742a6686] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 234ea45 >2012-12-17 10:00:53,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [742a6686] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:53,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [742a6686] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 234ea45 >2012-12-17 10:00:54,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:00:54,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5025 >2012-12-17 10:00:54,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5025, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28db5859 >2012-12-17 10:00:54,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling started: taskId = af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e >2012-12-17 10:00:55,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended: taskId = af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e task status = finished >2012-12-17 10:00:55,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended, spm status: SPM >2012-12-17 10:00:55,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=af6e4c20-ea02-4d0e-8f1f-91f3f6857e2e), log id: 1befdf1c >2012-12-17 10:00:55,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMClearTaskVDSCommand, log id: 1befdf1c >2012-12-17 10:00:55,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2120b598, log id: 28db5859 >2012-12-17 10:00:55,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:00:55,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:55,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 187dc720 >2012-12-17 10:00:55,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 187dc720 >2012-12-17 10:00:55,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1421d86b >2012-12-17 10:00:55,608 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:00:55,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 699c126f >2012-12-17 10:00:55,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:00:55,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135ab894 >2012-12-17 10:00:55,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135ab894 >2012-12-17 10:00:55,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 699c126f >2012-12-17 10:00:55,628 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:01:03,370 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:01:04,388 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [44b78d4a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:01:04,760 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [569d76f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:01:04,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6753cb6d >2012-12-17 10:01:04,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5bdf9c31 >2012-12-17 10:01:04,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:05,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SpmStopVDSCommand, log id: 5bdf9c31 >2012-12-17 10:01:05,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, ResetIrsVDSCommand, log id: 6753cb6d >2012-12-17 10:01:05,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 450d893c >2012-12-17 10:01:05,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:05,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 450d893c >2012-12-17 10:01:13,936 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4bbb254] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:01:15,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:01:15,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5026 >2012-12-17 10:01:15,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5026, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1c479c82 >2012-12-17 10:01:15,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling started: taskId = aed7acca-67de-4d97-8e35-a018673077dc >2012-12-17 10:01:16,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended: taskId = aed7acca-67de-4d97-8e35-a018673077dc task status = finished >2012-12-17 10:01:16,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] spmStart polling ended, spm status: SPM >2012-12-17 10:01:16,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aed7acca-67de-4d97-8e35-a018673077dc), log id: 5b925b8c >2012-12-17 10:01:16,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, HSMClearTaskVDSCommand, log id: 5b925b8c >2012-12-17 10:01:16,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [3e0f3441] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65b8734b, log id: 1c479c82 >2012-12-17 10:01:17,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:01:17,254 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:01:17,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7223f2cc >2012-12-17 10:01:17,271 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [3e0f3441] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:01:17,423 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [53c52fa0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:01:17,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2335a941 >2012-12-17 10:01:17,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f493480 >2012-12-17 10:01:17,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:17,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, SpmStopVDSCommand, log id: 4f493480 >2012-12-17 10:01:17,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, ResetIrsVDSCommand, log id: 2335a941 >2012-12-17 10:01:17,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15bd1d2f >2012-12-17 10:01:17,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:17,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15bd1d2f >2012-12-17 10:01:17,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:01:17,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5027 >2012-12-17 10:01:17,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5027, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 537babf6 >2012-12-17 10:01:17,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling started: taskId = d8a398fe-8f71-4455-8a4e-1f4b66c496ec >2012-12-17 10:01:18,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling ended: taskId = d8a398fe-8f71-4455-8a4e-1f4b66c496ec task status = finished >2012-12-17 10:01:18,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] spmStart polling ended, spm status: SPM >2012-12-17 10:01:18,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8a398fe-8f71-4455-8a4e-1f4b66c496ec), log id: 736b7147 >2012-12-17 10:01:18,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMClearTaskVDSCommand, log id: 736b7147 >2012-12-17 10:01:18,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@12da2784, log id: 537babf6 >2012-12-17 10:01:18,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:01:18,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:18,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 626b1671 >2012-12-17 10:01:19,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 626b1671 >2012-12-17 10:01:19,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7223f2cc >2012-12-17 10:01:19,001 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:01:19,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 494dd58a >2012-12-17 10:01:19,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:19,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 35ef4b77 >2012-12-17 10:01:19,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 35ef4b77 >2012-12-17 10:01:19,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [6827474] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 494dd58a >2012-12-17 10:01:19,022 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [6827474] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:01:24,374 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:01:27,875 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4ea30a59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:01:28,540 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [5cec762d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:01:28,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: da67bef >2012-12-17 10:01:28,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e317ef6 >2012-12-17 10:01:28,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:28,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, SpmStopVDSCommand, log id: 7e317ef6 >2012-12-17 10:01:28,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, ResetIrsVDSCommand, log id: da67bef >2012-12-17 10:01:29,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 547b1cdb >2012-12-17 10:01:29,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5cec762d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:29,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5cec762d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 547b1cdb >2012-12-17 10:01:34,843 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:01:39,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:01:39,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5028 >2012-12-17 10:01:39,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5028, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6947245b >2012-12-17 10:01:39,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling started: taskId = d8464e7c-9194-4e3c-83a3-79253d717741 >2012-12-17 10:01:40,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling ended: taskId = d8464e7c-9194-4e3c-83a3-79253d717741 task status = finished >2012-12-17 10:01:40,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] spmStart polling ended, spm status: SPM >2012-12-17 10:01:40,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8464e7c-9194-4e3c-83a3-79253d717741), log id: 74e245b9 >2012-12-17 10:01:40,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, HSMClearTaskVDSCommand, log id: 74e245b9 >2012-12-17 10:01:40,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [854c8c6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@334fb0a2, log id: 6947245b >2012-12-17 10:01:40,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:01:40,475 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:01:40,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 696aabdb >2012-12-17 10:01:40,487 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [854c8c6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:01:40,875 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [75114a28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:01:42,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6613c46a >2012-12-17 10:01:42,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d89812d >2012-12-17 10:01:42,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:42,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, SpmStopVDSCommand, log id: 5d89812d >2012-12-17 10:01:42,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, ResetIrsVDSCommand, log id: 6613c46a >2012-12-17 10:01:42,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c2ea578 >2012-12-17 10:01:42,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:42,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c2ea578 >2012-12-17 10:01:43,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:01:43,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5029 >2012-12-17 10:01:43,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5029, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d14a10b >2012-12-17 10:01:43,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling started: taskId = 69f0ce72-5f18-477b-9771-41c0a269bbc6 >2012-12-17 10:01:44,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling ended: taskId = 69f0ce72-5f18-477b-9771-41c0a269bbc6 task status = finished >2012-12-17 10:01:44,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] spmStart polling ended, spm status: SPM >2012-12-17 10:01:44,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=69f0ce72-5f18-477b-9771-41c0a269bbc6), log id: 77f7b1e4 >2012-12-17 10:01:44,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMClearTaskVDSCommand, log id: 77f7b1e4 >2012-12-17 10:01:44,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26c02d44, log id: d14a10b >2012-12-17 10:01:44,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:01:44,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:44,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d2fe023 >2012-12-17 10:01:44,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d2fe023 >2012-12-17 10:01:44,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 696aabdb >2012-12-17 10:01:44,234 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:01:44,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 63f2c472 >2012-12-17 10:01:44,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:44,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7981023f >2012-12-17 10:01:44,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7981023f >2012-12-17 10:01:44,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-23) [140420a5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 63f2c472 >2012-12-17 10:01:44,255 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-23) [140420a5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:01:45,258 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:01:53,114 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7712cc49] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:01:53,555 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:01:53,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69f85066 >2012-12-17 10:01:53,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 600daa0a >2012-12-17 10:01:53,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:54,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, SpmStopVDSCommand, log id: 600daa0a >2012-12-17 10:01:54,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, ResetIrsVDSCommand, log id: 69f85066 >2012-12-17 10:01:54,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e81ec2c >2012-12-17 10:01:54,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:01:54,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [39a6c2d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e81ec2c >2012-12-17 10:01:55,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:02:04,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:04,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5030 >2012-12-17 10:02:04,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5030, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5557de58 >2012-12-17 10:02:04,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling started: taskId = ec0535f4-e1ce-4fda-a34e-228d48f1f0f6 >2012-12-17 10:02:05,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling ended: taskId = ec0535f4-e1ce-4fda-a34e-228d48f1f0f6 task status = finished >2012-12-17 10:02:05,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] spmStart polling ended, spm status: SPM >2012-12-17 10:02:05,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [23663da4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ec0535f4-e1ce-4fda-a34e-228d48f1f0f6), log id: 25047d0b >2012-12-17 10:02:05,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, HSMClearTaskVDSCommand, log id: 25047d0b >2012-12-17 10:02:05,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [23663da4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@49ad7029, log id: 5557de58 >2012-12-17 10:02:05,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:08,287 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:02:08,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47af4761 >2012-12-17 10:02:08,287 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [23663da4] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:02:09,654 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [26a9278b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:02:09,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 24a67139 >2012-12-17 10:02:09,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 49cf5dbe >2012-12-17 10:02:09,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:10,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStopVDSCommand, log id: 49cf5dbe >2012-12-17 10:02:10,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, ResetIrsVDSCommand, log id: 24a67139 >2012-12-17 10:02:10,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35f3ec9e >2012-12-17 10:02:10,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:10,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35f3ec9e >2012-12-17 10:02:10,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:10,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5031 >2012-12-17 10:02:10,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5031, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a3b680b >2012-12-17 10:02:10,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling started: taskId = 4a6dad52-bf6f-4474-924b-d432faeef8af >2012-12-17 10:02:11,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended: taskId = 4a6dad52-bf6f-4474-924b-d432faeef8af task status = finished >2012-12-17 10:02:11,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended, spm status: SPM >2012-12-17 10:02:11,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a6dad52-bf6f-4474-924b-d432faeef8af), log id: 12a11d42 >2012-12-17 10:02:11,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMClearTaskVDSCommand, log id: 12a11d42 >2012-12-17 10:02:11,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f87e19c, log id: a3b680b >2012-12-17 10:02:11,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:11,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:11,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6a2bacc2 >2012-12-17 10:02:12,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6a2bacc2 >2012-12-17 10:02:12,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 47af4761 >2012-12-17 10:02:12,013 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:02:12,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 595b2b02 >2012-12-17 10:02:12,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:12,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f7607f9 >2012-12-17 10:02:12,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f7607f9 >2012-12-17 10:02:12,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 595b2b02 >2012-12-17 10:02:12,035 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:02:18,994 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:02:20,395 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5cec762d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:02:21,841 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [3ced8e0c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:02:22,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f322719 >2012-12-17 10:02:22,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 48e63073 >2012-12-17 10:02:22,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:22,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, SpmStopVDSCommand, log id: 48e63073 >2012-12-17 10:02:22,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, ResetIrsVDSCommand, log id: 3f322719 >2012-12-17 10:02:22,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75379658 >2012-12-17 10:02:22,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [3ced8e0c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:22,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [3ced8e0c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75379658 >2012-12-17 10:02:29,343 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:02:32,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:32,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5032 >2012-12-17 10:02:32,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5032, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4526c1b3 >2012-12-17 10:02:32,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling started: taskId = dc4e42b1-03b3-41cf-9ecd-27fb7dced61c >2012-12-17 10:02:33,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended: taskId = dc4e42b1-03b3-41cf-9ecd-27fb7dced61c task status = finished >2012-12-17 10:02:33,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] spmStart polling ended, spm status: SPM >2012-12-17 10:02:33,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc4e42b1-03b3-41cf-9ecd-27fb7dced61c), log id: 7c3b4e74 >2012-12-17 10:02:33,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, HSMClearTaskVDSCommand, log id: 7c3b4e74 >2012-12-17 10:02:33,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [127bfc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199be164, log id: 4526c1b3 >2012-12-17 10:02:33,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:33,822 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:02:33,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1424dc66 >2012-12-17 10:02:33,875 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [127bfc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:02:34,335 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [5920de03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:02:34,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 227b470a >2012-12-17 10:02:34,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3496d411 >2012-12-17 10:02:34,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:34,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SpmStopVDSCommand, log id: 3496d411 >2012-12-17 10:02:34,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, ResetIrsVDSCommand, log id: 227b470a >2012-12-17 10:02:34,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 636a4c7e >2012-12-17 10:02:34,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:34,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 636a4c7e >2012-12-17 10:02:34,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:34,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5033 >2012-12-17 10:02:34,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5033, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 60d5ba0c >2012-12-17 10:02:34,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c >2012-12-17 10:02:35,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c task status = finished >2012-12-17 10:02:35,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM >2012-12-17 10:02:35,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=21a8037b-a5d0-4ad2-a5db-6cd521c8dd0c), log id: 7130ba21 >2012-12-17 10:02:35,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 7130ba21 >2012-12-17 10:02:35,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@360c0695, log id: 60d5ba0c >2012-12-17 10:02:36,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:36,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:36,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60411f57 >2012-12-17 10:02:36,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60411f57 >2012-12-17 10:02:36,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1424dc66 >2012-12-17 10:02:36,898 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:02:36,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: dcb7405 >2012-12-17 10:02:36,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:36,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4ee22466 >2012-12-17 10:02:36,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4ee22466 >2012-12-17 10:02:36,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: dcb7405 >2012-12-17 10:02:36,920 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:02:39,627 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:02:44,927 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [29b31be1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:02:45,030 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [4a8b829d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:02:45,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2bdeffe6 >2012-12-17 10:02:45,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5d20e91a >2012-12-17 10:02:45,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:45,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, SpmStopVDSCommand, log id: 5d20e91a >2012-12-17 10:02:45,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, ResetIrsVDSCommand, log id: 2bdeffe6 >2012-12-17 10:02:45,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: dd25436 >2012-12-17 10:02:45,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:45,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: dd25436 >2012-12-17 10:02:49,962 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:02:55,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:55,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5034 >2012-12-17 10:02:55,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5034, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 786b6820 >2012-12-17 10:02:55,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling started: taskId = bc40b477-fe1b-4827-af75-b6c54b5963ba >2012-12-17 10:02:56,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling ended: taskId = bc40b477-fe1b-4827-af75-b6c54b5963ba task status = finished >2012-12-17 10:02:56,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] spmStart polling ended, spm status: SPM >2012-12-17 10:02:56,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bc40b477-fe1b-4827-af75-b6c54b5963ba), log id: 49e4b11c >2012-12-17 10:02:56,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, HSMClearTaskVDSCommand, log id: 49e4b11c >2012-12-17 10:02:56,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [53c52fa0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4527cb2a, log id: 786b6820 >2012-12-17 10:02:57,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:57,452 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:02:57,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c25ec6f >2012-12-17 10:02:57,453 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [53c52fa0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:02:57,803 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:02:57,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 28577e39 >2012-12-17 10:02:57,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 40027332 >2012-12-17 10:02:58,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:58,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, SpmStopVDSCommand, log id: 40027332 >2012-12-17 10:02:58,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, ResetIrsVDSCommand, log id: 28577e39 >2012-12-17 10:02:58,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e0f4c4d >2012-12-17 10:02:58,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:58,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e0f4c4d >2012-12-17 10:02:58,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:02:58,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5035 >2012-12-17 10:02:58,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5035, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 561f9b9f >2012-12-17 10:02:58,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling started: taskId = 14e9429e-dc0b-40f4-a320-cf27d181752e >2012-12-17 10:02:59,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended: taskId = 14e9429e-dc0b-40f4-a320-cf27d181752e task status = finished >2012-12-17 10:02:59,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] spmStart polling ended, spm status: SPM >2012-12-17 10:02:59,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=14e9429e-dc0b-40f4-a320-cf27d181752e), log id: 4d165fe3 >2012-12-17 10:02:59,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMClearTaskVDSCommand, log id: 4d165fe3 >2012-12-17 10:02:59,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77f1579e, log id: 561f9b9f >2012-12-17 10:02:59,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:02:59,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:59,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42ef9766 >2012-12-17 10:02:59,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42ef9766 >2012-12-17 10:02:59,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c25ec6f >2012-12-17 10:02:59,623 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:02:59,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7fbccbde >2012-12-17 10:02:59,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:02:59,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3c467537 >2012-12-17 10:02:59,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c467537 >2012-12-17 10:02:59,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-20) [56dc6659] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fbccbde >2012-12-17 10:02:59,643 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-20) [56dc6659] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:03:00,276 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:03:08,422 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [16b67351] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:03:08,771 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [7dff3777] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:03:09,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 570ededb >2012-12-17 10:03:09,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d968357 >2012-12-17 10:03:09,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:09,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SpmStopVDSCommand, log id: 6d968357 >2012-12-17 10:03:09,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, ResetIrsVDSCommand, log id: 570ededb >2012-12-17 10:03:09,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3cdc8682 >2012-12-17 10:03:09,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:09,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3cdc8682 >2012-12-17 10:03:10,650 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:03:19,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:03:19,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5036 >2012-12-17 10:03:19,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5036, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65ffd80d >2012-12-17 10:03:19,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling started: taskId = 7926acf1-8885-40a0-b9df-a87544371ae7 >2012-12-17 10:03:21,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling ended: taskId = 7926acf1-8885-40a0-b9df-a87544371ae7 task status = finished >2012-12-17 10:03:21,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] spmStart polling ended, spm status: SPM >2012-12-17 10:03:21,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5b398282] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7926acf1-8885-40a0-b9df-a87544371ae7), log id: 9b58fe1 >2012-12-17 10:03:21,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, HSMClearTaskVDSCommand, log id: 9b58fe1 >2012-12-17 10:03:21,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5b398282] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ef901c1, log id: 65ffd80d >2012-12-17 10:03:21,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:03:21,990 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:03:21,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 16a169a3 >2012-12-17 10:03:22,165 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5b398282] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:03:22,625 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [62ee0772] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:03:22,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 65ca6b14 >2012-12-17 10:03:22,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e146a85 >2012-12-17 10:03:22,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:23,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, SpmStopVDSCommand, log id: 2e146a85 >2012-12-17 10:03:23,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, ResetIrsVDSCommand, log id: 65ca6b14 >2012-12-17 10:03:23,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31d70570 >2012-12-17 10:03:23,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:23,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31d70570 >2012-12-17 10:03:23,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:03:23,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5037 >2012-12-17 10:03:23,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5037, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69fc1c91 >2012-12-17 10:03:23,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling started: taskId = aad38a30-e888-4ed0-945c-33c4bb844f83 >2012-12-17 10:03:24,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling ended: taskId = aad38a30-e888-4ed0-945c-33c4bb844f83 task status = finished >2012-12-17 10:03:24,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] spmStart polling ended, spm status: SPM >2012-12-17 10:03:24,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aad38a30-e888-4ed0-945c-33c4bb844f83), log id: 6c0e61e1 >2012-12-17 10:03:24,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMClearTaskVDSCommand, log id: 6c0e61e1 >2012-12-17 10:03:24,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@243c3233, log id: 69fc1c91 >2012-12-17 10:03:24,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:03:24,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:24,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 746983f0 >2012-12-17 10:03:24,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 746983f0 >2012-12-17 10:03:24,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 16a169a3 >2012-12-17 10:03:24,713 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:03:24,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b553d45 >2012-12-17 10:03:24,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:24,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 796072b7 >2012-12-17 10:03:24,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 796072b7 >2012-12-17 10:03:24,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-73) [786860d5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b553d45 >2012-12-17 10:03:24,732 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-73) [786860d5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:03:32,268 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:03:33,795 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [193f8ef] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:03:34,187 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [53480d94] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:03:34,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: eadacb1 >2012-12-17 10:03:34,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 481a6b49 >2012-12-17 10:03:34,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:34,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SpmStopVDSCommand, log id: 481a6b49 >2012-12-17 10:03:34,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, ResetIrsVDSCommand, log id: eadacb1 >2012-12-17 10:03:34,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5812c372 >2012-12-17 10:03:34,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:34,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5812c372 >2012-12-17 10:03:42,583 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:03:44,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:03:45,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5038 >2012-12-17 10:03:45,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5038, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7cc0282d >2012-12-17 10:03:45,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling started: taskId = 72ce5fb8-b929-4316-8407-bd9b0b4bec63 >2012-12-17 10:03:46,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended: taskId = 72ce5fb8-b929-4316-8407-bd9b0b4bec63 task status = finished >2012-12-17 10:03:46,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] spmStart polling ended, spm status: SPM >2012-12-17 10:03:46,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72ce5fb8-b929-4316-8407-bd9b0b4bec63), log id: 30fa6f76 >2012-12-17 10:03:46,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, HSMClearTaskVDSCommand, log id: 30fa6f76 >2012-12-17 10:03:46,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [604cecb9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9abc519, log id: 7cc0282d >2012-12-17 10:03:46,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:03:47,277 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:03:47,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a5a2ded >2012-12-17 10:03:47,304 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [604cecb9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:03:47,809 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [2f07235e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:03:47,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a8fc6ba >2012-12-17 10:03:47,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 35ba84ec >2012-12-17 10:03:47,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:48,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, SpmStopVDSCommand, log id: 35ba84ec >2012-12-17 10:03:48,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, ResetIrsVDSCommand, log id: 1a8fc6ba >2012-12-17 10:03:48,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6596308f >2012-12-17 10:03:48,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [2f07235e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:48,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [2f07235e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6596308f >2012-12-17 10:03:48,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:03:48,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5039 >2012-12-17 10:03:48,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5039, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f44f41e >2012-12-17 10:03:48,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = a248f873-dd14-448a-9108-eb4c4598959d >2012-12-17 10:03:49,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = a248f873-dd14-448a-9108-eb4c4598959d task status = finished >2012-12-17 10:03:49,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM >2012-12-17 10:03:49,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a248f873-dd14-448a-9108-eb4c4598959d), log id: 6c3b8a1f >2012-12-17 10:03:49,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 6c3b8a1f >2012-12-17 10:03:49,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38e930bc, log id: 5f44f41e >2012-12-17 10:03:49,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:03:49,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:49,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 222b68a1 >2012-12-17 10:03:49,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 222b68a1 >2012-12-17 10:03:49,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1a5a2ded >2012-12-17 10:03:49,542 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:03:49,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fe2338d >2012-12-17 10:03:49,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:49,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3b6a1a93 >2012-12-17 10:03:49,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b6a1a93 >2012-12-17 10:03:49,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fe2338d >2012-12-17 10:03:49,563 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-4) [5fb78147] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:03:53,010 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [38047437] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:03:58,328 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1dafbc5c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:03:58,683 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [4b390d42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:03:58,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 420c58ce >2012-12-17 10:03:58,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 488f09d5 >2012-12-17 10:03:58,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:58,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, SpmStopVDSCommand, log id: 488f09d5 >2012-12-17 10:03:59,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, ResetIrsVDSCommand, log id: 420c58ce >2012-12-17 10:03:59,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7efb6aee >2012-12-17 10:03:59,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [4b390d42] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:03:59,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [4b390d42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7efb6aee >2012-12-17 10:04:03,308 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:04:09,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:04:09,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5040 >2012-12-17 10:04:09,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5040, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21b95bce >2012-12-17 10:04:09,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling started: taskId = aef60ccc-5f7d-4839-85fd-c3d82b093aae >2012-12-17 10:04:10,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling ended: taskId = aef60ccc-5f7d-4839-85fd-c3d82b093aae task status = finished >2012-12-17 10:04:10,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] spmStart polling ended, spm status: SPM >2012-12-17 10:04:10,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [75114a28] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=aef60ccc-5f7d-4839-85fd-c3d82b093aae), log id: 59e6eed8 >2012-12-17 10:04:10,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, HSMClearTaskVDSCommand, log id: 59e6eed8 >2012-12-17 10:04:10,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [75114a28] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@484ad504, log id: 21b95bce >2012-12-17 10:04:10,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:04:10,253 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:04:10,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 40437d43 >2012-12-17 10:04:10,286 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [75114a28] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:04:11,838 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [10600189] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:04:12,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25b58411 >2012-12-17 10:04:12,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ed6122 >2012-12-17 10:04:12,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:12,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStopVDSCommand, log id: 4ed6122 >2012-12-17 10:04:12,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, ResetIrsVDSCommand, log id: 25b58411 >2012-12-17 10:04:12,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 385241f4 >2012-12-17 10:04:12,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:12,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 385241f4 >2012-12-17 10:04:12,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:04:12,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5041 >2012-12-17 10:04:12,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5041, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71628853 >2012-12-17 10:04:12,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling started: taskId = c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4 >2012-12-17 10:04:13,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling ended: taskId = c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4 task status = finished >2012-12-17 10:04:13,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] spmStart polling ended, spm status: SPM >2012-12-17 10:04:13,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c6dae77d-e1a6-4ffa-bc3c-d49a2a8c59c4), log id: 5ea03201 >2012-12-17 10:04:13,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMClearTaskVDSCommand, log id: 5ea03201 >2012-12-17 10:04:13,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6733102f, log id: 71628853 >2012-12-17 10:04:14,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:04:14,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:14,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65ef10bc >2012-12-17 10:04:14,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65ef10bc >2012-12-17 10:04:14,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40437d43 >2012-12-17 10:04:14,052 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:04:14,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3233daec >2012-12-17 10:04:14,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:14,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5d760880 >2012-12-17 10:04:14,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5d760880 >2012-12-17 10:04:14,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [5920de03] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3233daec >2012-12-17 10:04:14,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [5920de03] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:04:22,970 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [462eb8a5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:04:23,110 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [3c3f7918] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:04:23,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 649efde7 >2012-12-17 10:04:23,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4ddf3d72 >2012-12-17 10:04:23,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:23,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, SpmStopVDSCommand, log id: 4ddf3d72 >2012-12-17 10:04:23,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, ResetIrsVDSCommand, log id: 649efde7 >2012-12-17 10:04:23,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20ee4f5a >2012-12-17 10:04:23,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [3c3f7918] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:23,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [3c3f7918] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20ee4f5a >2012-12-17 10:04:23,799 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:04:33,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:04:33,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5042 >2012-12-17 10:04:33,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5042, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 584f8a32 >2012-12-17 10:04:33,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling started: taskId = 9728bcdc-475d-4a21-9b07-3d9b701b8d90 >2012-12-17 10:04:34,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling ended: taskId = 9728bcdc-475d-4a21-9b07-3d9b701b8d90 task status = finished >2012-12-17 10:04:34,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] spmStart polling ended, spm status: SPM >2012-12-17 10:04:34,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9728bcdc-475d-4a21-9b07-3d9b701b8d90), log id: 503d1f56 >2012-12-17 10:04:34,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, HSMClearTaskVDSCommand, log id: 503d1f56 >2012-12-17 10:04:34,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [6edf3aeb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5cd47041, log id: 584f8a32 >2012-12-17 10:04:34,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:04:34,804 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:04:34,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2e486d93 >2012-12-17 10:04:34,844 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [6edf3aeb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:04:35,139 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [42fa3e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:04:35,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13030497 >2012-12-17 10:04:35,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: ff7d99c >2012-12-17 10:04:35,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:35,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SpmStopVDSCommand, log id: ff7d99c >2012-12-17 10:04:37,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, ResetIrsVDSCommand, log id: 13030497 >2012-12-17 10:04:37,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b95e719 >2012-12-17 10:04:37,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:37,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b95e719 >2012-12-17 10:04:37,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:04:37,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5043 >2012-12-17 10:04:37,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5043, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ae1a24b >2012-12-17 10:04:37,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling started: taskId = f7d5661c-331a-4171-8d37-879254475f3b >2012-12-17 10:04:38,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended: taskId = f7d5661c-331a-4171-8d37-879254475f3b task status = finished >2012-12-17 10:04:38,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended, spm status: SPM >2012-12-17 10:04:38,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f7d5661c-331a-4171-8d37-879254475f3b), log id: 3e3b8b5c >2012-12-17 10:04:38,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMClearTaskVDSCommand, log id: 3e3b8b5c >2012-12-17 10:04:38,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@373a64ef, log id: 2ae1a24b >2012-12-17 10:04:38,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:04:38,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:38,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43790b0f >2012-12-17 10:04:38,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43790b0f >2012-12-17 10:04:38,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e486d93 >2012-12-17 10:04:38,670 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:04:38,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fc52864 >2012-12-17 10:04:38,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:38,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 26bb4133 >2012-12-17 10:04:38,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 26bb4133 >2012-12-17 10:04:38,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3fc52864 >2012-12-17 10:04:38,690 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:04:44,381 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:04:47,450 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [4b390d42] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:04:48,053 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [71a21411] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:04:48,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6f1669bf >2012-12-17 10:04:48,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 565c1d46 >2012-12-17 10:04:48,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:48,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, SpmStopVDSCommand, log id: 565c1d46 >2012-12-17 10:04:48,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, ResetIrsVDSCommand, log id: 6f1669bf >2012-12-17 10:04:48,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6de14679 >2012-12-17 10:04:48,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:04:48,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6de14679 >2012-12-17 10:04:54,686 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:04:58,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:04:58,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5044 >2012-12-17 10:04:58,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5044, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b1605a6 >2012-12-17 10:04:58,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling started: taskId = cf1054cc-d18c-4791-9f3e-12b621f064ed >2012-12-17 10:04:59,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling ended: taskId = cf1054cc-d18c-4791-9f3e-12b621f064ed task status = finished >2012-12-17 10:04:59,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] spmStart polling ended, spm status: SPM >2012-12-17 10:04:59,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=cf1054cc-d18c-4791-9f3e-12b621f064ed), log id: 208e1d14 >2012-12-17 10:04:59,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, HSMClearTaskVDSCommand, log id: 208e1d14 >2012-12-17 10:04:59,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [62ee0772] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52171cbe, log id: 6b1605a6 >2012-12-17 10:04:59,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:04:59,874 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:04:59,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 347abb03 >2012-12-17 10:04:59,898 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [62ee0772] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts >2012-12-17 10:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering 0 hosts >2012-12-17 10:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Checking autorecoverable hosts done >2012-12-17 10:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [140420a5] Autorecovering storage domains is disabled, skipping >2012-12-17 10:05:00,021 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [3d52876a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:00,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2232d05a >2012-12-17 10:05:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3eb85cb9 >2012-12-17 10:05:00,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, SpmStopVDSCommand, log id: 3eb85cb9 >2012-12-17 10:05:00,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, ResetIrsVDSCommand, log id: 2232d05a >2012-12-17 10:05:00,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 785986ee >2012-12-17 10:05:00,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3d52876a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:00,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [3d52876a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 785986ee >2012-12-17 10:05:00,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:05:00,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5045 >2012-12-17 10:05:00,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5045, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 701101df >2012-12-17 10:05:00,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling started: taskId = c760556c-239f-4fb5-9969-3c164b95adb4 >2012-12-17 10:05:01,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended: taskId = c760556c-239f-4fb5-9969-3c164b95adb4 task status = finished >2012-12-17 10:05:01,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended, spm status: SPM >2012-12-17 10:05:01,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c760556c-239f-4fb5-9969-3c164b95adb4), log id: 5e42bbf9 >2012-12-17 10:05:01,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMClearTaskVDSCommand, log id: 5e42bbf9 >2012-12-17 10:05:01,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f4308c7, log id: 701101df >2012-12-17 10:05:01,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:05:01,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:01,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 397b5197 >2012-12-17 10:05:01,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 397b5197 >2012-12-17 10:05:01,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 347abb03 >2012-12-17 10:05:01,706 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:01,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f29adfc >2012-12-17 10:05:01,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:01,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: bf5e032 >2012-12-17 10:05:01,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: bf5e032 >2012-12-17 10:05:01,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4f29adfc >2012-12-17 10:05:01,726 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-83) [10600189] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:04,908 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [69818e3e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:05:10,390 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [3c3f7918] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:10,543 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [8b964cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:10,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f974438 >2012-12-17 10:05:10,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 490255cd >2012-12-17 10:05:10,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:10,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, SpmStopVDSCommand, log id: 490255cd >2012-12-17 10:05:10,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, ResetIrsVDSCommand, log id: 4f974438 >2012-12-17 10:05:10,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 678f7f41 >2012-12-17 10:05:10,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:10,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 678f7f41 >2012-12-17 10:05:15,126 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:05:21,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:05:21,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5046 >2012-12-17 10:05:21,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5046, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f04bdd1 >2012-12-17 10:05:21,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling started: taskId = 4fd3b1cf-950e-49a9-ba67-c67f136b833b >2012-12-17 10:05:22,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling ended: taskId = 4fd3b1cf-950e-49a9-ba67-c67f136b833b task status = finished >2012-12-17 10:05:22,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] spmStart polling ended, spm status: SPM >2012-12-17 10:05:22,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4fd3b1cf-950e-49a9-ba67-c67f136b833b), log id: 2bcf0e67 >2012-12-17 10:05:22,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, HSMClearTaskVDSCommand, log id: 2bcf0e67 >2012-12-17 10:05:22,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [1b349a52] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26baa2a9, log id: 3f04bdd1 >2012-12-17 10:05:22,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:05:22,647 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:05:22,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69767428 >2012-12-17 10:05:22,736 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [1b349a52] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:22,876 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [b0e3bca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:22,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3d4ed90a >2012-12-17 10:05:22,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 46149bc6 >2012-12-17 10:05:22,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:23,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SpmStopVDSCommand, log id: 46149bc6 >2012-12-17 10:05:23,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, ResetIrsVDSCommand, log id: 3d4ed90a >2012-12-17 10:05:23,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 319a9f16 >2012-12-17 10:05:23,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:23,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 319a9f16 >2012-12-17 10:05:23,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:05:23,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5047 >2012-12-17 10:05:23,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5047, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74106d05 >2012-12-17 10:05:23,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling started: taskId = 1828dcec-a632-4bce-95b4-2c0ef99be33d >2012-12-17 10:05:24,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling ended: taskId = 1828dcec-a632-4bce-95b4-2c0ef99be33d task status = finished >2012-12-17 10:05:24,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] spmStart polling ended, spm status: SPM >2012-12-17 10:05:24,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1828dcec-a632-4bce-95b4-2c0ef99be33d), log id: 32963dab >2012-12-17 10:05:24,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMClearTaskVDSCommand, log id: 32963dab >2012-12-17 10:05:24,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@25aabf7f, log id: 74106d05 >2012-12-17 10:05:24,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:05:24,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:24,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1fc586f4 >2012-12-17 10:05:24,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1fc586f4 >2012-12-17 10:05:24,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69767428 >2012-12-17 10:05:24,624 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:24,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3dad3171 >2012-12-17 10:05:24,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:24,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3d900aa2 >2012-12-17 10:05:24,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3d900aa2 >2012-12-17 10:05:24,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-99) [20d09da7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dad3171 >2012-12-17 10:05:24,644 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-99) [20d09da7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:25,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:05:33,501 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [415bb057] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:33,613 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [464dfbdd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:33,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 53f45987 >2012-12-17 10:05:33,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1aaeda01 >2012-12-17 10:05:33,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:34,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, SpmStopVDSCommand, log id: 1aaeda01 >2012-12-17 10:05:34,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, ResetIrsVDSCommand, log id: 53f45987 >2012-12-17 10:05:34,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d02ac9e >2012-12-17 10:05:34,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:34,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d02ac9e >2012-12-17 10:05:35,501 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:05:44,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:05:44,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5048 >2012-12-17 10:05:44,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5048, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 284c17aa >2012-12-17 10:05:44,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling started: taskId = 72c63a02-5d82-4fd1-8067-bc5dae55bc8b >2012-12-17 10:05:45,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling ended: taskId = 72c63a02-5d82-4fd1-8067-bc5dae55bc8b task status = finished >2012-12-17 10:05:45,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] spmStart polling ended, spm status: SPM >2012-12-17 10:05:45,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=72c63a02-5d82-4fd1-8067-bc5dae55bc8b), log id: 340fb222 >2012-12-17 10:05:45,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, HSMClearTaskVDSCommand, log id: 340fb222 >2012-12-17 10:05:45,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7277f0e1, log id: 284c17aa >2012-12-17 10:05:45,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:05:45,548 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:05:45,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a2ac029 >2012-12-17 10:05:45,567 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [1fcd2a4c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:45,789 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [375e73d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:45,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 661ac216 >2012-12-17 10:05:45,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 190daf6b >2012-12-17 10:05:45,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:46,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, SpmStopVDSCommand, log id: 190daf6b >2012-12-17 10:05:46,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, ResetIrsVDSCommand, log id: 661ac216 >2012-12-17 10:05:46,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f384a39 >2012-12-17 10:05:46,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [375e73d3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:46,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [375e73d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f384a39 >2012-12-17 10:05:46,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:05:46,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5049 >2012-12-17 10:05:46,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5049, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 756e85f5 >2012-12-17 10:05:46,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = 8cf4c681-55f2-4315-b0b0-24f756854e0b >2012-12-17 10:05:47,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = 8cf4c681-55f2-4315-b0b0-24f756854e0b task status = finished >2012-12-17 10:05:47,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM >2012-12-17 10:05:47,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8cf4c681-55f2-4315-b0b0-24f756854e0b), log id: 7916c3d5 >2012-12-17 10:05:47,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 7916c3d5 >2012-12-17 10:05:47,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4593130e, log id: 756e85f5 >2012-12-17 10:05:47,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:05:47,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:47,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72fe1098 >2012-12-17 10:05:47,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72fe1098 >2012-12-17 10:05:47,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6a2ac029 >2012-12-17 10:05:47,912 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:47,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3f2046e2 >2012-12-17 10:05:47,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:47,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3cb2457a >2012-12-17 10:05:47,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3cb2457a >2012-12-17 10:05:47,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f2046e2 >2012-12-17 10:05:47,932 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:05:56,908 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4549d797] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:05:56,909 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:05:57,074 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:05:57,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: a3f922c >2012-12-17 10:05:57,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 636a1ed8 >2012-12-17 10:05:57,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:57,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, SpmStopVDSCommand, log id: 636a1ed8 >2012-12-17 10:05:57,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, ResetIrsVDSCommand, log id: a3f922c >2012-12-17 10:05:57,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1168a7ec >2012-12-17 10:05:57,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:05:57,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1168a7ec >2012-12-17 10:06:07,477 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:06:08,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:08,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5050 >2012-12-17 10:06:08,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5050, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c6eb586 >2012-12-17 10:06:08,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling started: taskId = bfbcf4d7-c4a1-4ddf-acab-7882a230856d >2012-12-17 10:06:09,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling ended: taskId = bfbcf4d7-c4a1-4ddf-acab-7882a230856d task status = finished >2012-12-17 10:06:09,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] spmStart polling ended, spm status: SPM >2012-12-17 10:06:09,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [21d8114] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bfbcf4d7-c4a1-4ddf-acab-7882a230856d), log id: 5d1737c1 >2012-12-17 10:06:09,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, HSMClearTaskVDSCommand, log id: 5d1737c1 >2012-12-17 10:06:09,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [21d8114] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f4d8fa6, log id: 6c6eb586 >2012-12-17 10:06:09,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:09,209 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:06:09,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13791afa >2012-12-17 10:06:09,228 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [21d8114] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:06:09,433 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [18826afe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:06:09,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 48f659b5 >2012-12-17 10:06:09,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3a91240d >2012-12-17 10:06:09,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:09,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, SpmStopVDSCommand, log id: 3a91240d >2012-12-17 10:06:09,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, ResetIrsVDSCommand, log id: 48f659b5 >2012-12-17 10:06:09,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [18826afe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ad3686f >2012-12-17 10:06:09,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:09,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [18826afe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ad3686f >2012-12-17 10:06:09,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:10,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5051 >2012-12-17 10:06:10,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5051, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2c401734 >2012-12-17 10:06:10,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling started: taskId = bcf00db9-5ff6-4438-8c6b-d90a767c61e3 >2012-12-17 10:06:11,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling ended: taskId = bcf00db9-5ff6-4438-8c6b-d90a767c61e3 task status = finished >2012-12-17 10:06:11,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] spmStart polling ended, spm status: SPM >2012-12-17 10:06:11,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bcf00db9-5ff6-4438-8c6b-d90a767c61e3), log id: 4d6e6ac1 >2012-12-17 10:06:11,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMClearTaskVDSCommand, log id: 4d6e6ac1 >2012-12-17 10:06:11,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c22cc9e, log id: 2c401734 >2012-12-17 10:06:11,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [24cb4742] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:11,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:11,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1972f3ae >2012-12-17 10:06:11,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1972f3ae >2012-12-17 10:06:11,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13791afa >2012-12-17 10:06:11,789 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:11,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49dc318f >2012-12-17 10:06:11,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:11,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 13c09d20 >2012-12-17 10:06:11,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 13c09d20 >2012-12-17 10:06:11,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-37) [24cb4742] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49dc318f >2012-12-17 10:06:11,810 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-37) [24cb4742] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:17,715 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:06:20,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [6a7edb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:06:20,174 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [729d4e04] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:06:20,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6ee8dc7 >2012-12-17 10:06:20,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2748c783 >2012-12-17 10:06:20,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:20,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, SpmStopVDSCommand, log id: 2748c783 >2012-12-17 10:06:20,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, ResetIrsVDSCommand, log id: 6ee8dc7 >2012-12-17 10:06:20,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e55f833 >2012-12-17 10:06:20,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [729d4e04] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:20,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [729d4e04] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e55f833 >2012-12-17 10:06:27,887 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:06:30,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:30,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5052 >2012-12-17 10:06:30,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5052, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 226188d7 >2012-12-17 10:06:30,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling started: taskId = 81c2f99a-953e-4a75-8d24-be2b4e87cf78 >2012-12-17 10:06:31,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling ended: taskId = 81c2f99a-953e-4a75-8d24-be2b4e87cf78 task status = finished >2012-12-17 10:06:31,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] spmStart polling ended, spm status: SPM >2012-12-17 10:06:31,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=81c2f99a-953e-4a75-8d24-be2b4e87cf78), log id: 57774480 >2012-12-17 10:06:31,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, HSMClearTaskVDSCommand, log id: 57774480 >2012-12-17 10:06:31,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [3c62925a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57325c5c, log id: 226188d7 >2012-12-17 10:06:31,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:31,830 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:06:31,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6816a4be >2012-12-17 10:06:31,850 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [3c62925a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:06:32,089 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:06:32,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 26c5d740 >2012-12-17 10:06:32,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 57da04a1 >2012-12-17 10:06:32,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:32,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, SpmStopVDSCommand, log id: 57da04a1 >2012-12-17 10:06:32,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, ResetIrsVDSCommand, log id: 26c5d740 >2012-12-17 10:06:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e1c41cd >2012-12-17 10:06:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:32,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [5cd90aa7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e1c41cd >2012-12-17 10:06:32,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:32,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5053 >2012-12-17 10:06:32,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5053, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46276be >2012-12-17 10:06:32,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = b835766e-e65d-45eb-89bc-b42f36159d0b >2012-12-17 10:06:33,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = b835766e-e65d-45eb-89bc-b42f36159d0b task status = finished >2012-12-17 10:06:33,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM >2012-12-17 10:06:33,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b835766e-e65d-45eb-89bc-b42f36159d0b), log id: 2a1658a9 >2012-12-17 10:06:33,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 2a1658a9 >2012-12-17 10:06:33,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2116d4d5, log id: 46276be >2012-12-17 10:06:33,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:33,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:33,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c1a1da6 >2012-12-17 10:06:33,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c1a1da6 >2012-12-17 10:06:33,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6816a4be >2012-12-17 10:06:33,589 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:33,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1dde98f3 >2012-12-17 10:06:33,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:33,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1462b6ca >2012-12-17 10:06:33,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1462b6ca >2012-12-17 10:06:33,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1dde98f3 >2012-12-17 10:06:33,609 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-1) [384b399c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:38,075 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:06:42,488 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [3e0f1a8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:06:42,589 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [75689c2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:06:42,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: c56637f >2012-12-17 10:06:42,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6fe5701e >2012-12-17 10:06:42,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:42,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStopVDSCommand, log id: 6fe5701e >2012-12-17 10:06:42,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, ResetIrsVDSCommand, log id: c56637f >2012-12-17 10:06:42,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27b3ea9d >2012-12-17 10:06:42,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:42,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27b3ea9d >2012-12-17 10:06:48,248 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:06:53,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:53,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5054 >2012-12-17 10:06:53,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5054, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d07078d >2012-12-17 10:06:53,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling started: taskId = 6682a771-d5b1-4cf6-9c07-7e6524d98ad5 >2012-12-17 10:06:54,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended: taskId = 6682a771-d5b1-4cf6-9c07-7e6524d98ad5 task status = finished >2012-12-17 10:06:54,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] spmStart polling ended, spm status: SPM >2012-12-17 10:06:54,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6682a771-d5b1-4cf6-9c07-7e6524d98ad5), log id: 56b962b1 >2012-12-17 10:06:54,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, HSMClearTaskVDSCommand, log id: 56b962b1 >2012-12-17 10:06:54,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [6eb2e264] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24fe7ef2, log id: 1d07078d >2012-12-17 10:06:54,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:54,186 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:06:54,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51ff7820 >2012-12-17 10:06:54,208 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6eb2e264] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:06:54,335 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [5074b866] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:06:54,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ad2bb40 >2012-12-17 10:06:54,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f79c013 >2012-12-17 10:06:54,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:54,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, SpmStopVDSCommand, log id: 5f79c013 >2012-12-17 10:06:54,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, ResetIrsVDSCommand, log id: 5ad2bb40 >2012-12-17 10:06:54,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [5074b866] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13d5f1d7 >2012-12-17 10:06:54,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [5074b866] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:54,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [5074b866] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13d5f1d7 >2012-12-17 10:06:54,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:06:54,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5055 >2012-12-17 10:06:54,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5055, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71fdb627 >2012-12-17 10:06:54,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling started: taskId = 589ac769-24ec-49fc-b320-9532618da2fa >2012-12-17 10:06:55,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended: taskId = 589ac769-24ec-49fc-b320-9532618da2fa task status = finished >2012-12-17 10:06:55,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] spmStart polling ended, spm status: SPM >2012-12-17 10:06:55,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=589ac769-24ec-49fc-b320-9532618da2fa), log id: 1eca421c >2012-12-17 10:06:55,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMClearTaskVDSCommand, log id: 1eca421c >2012-12-17 10:06:55,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70ba4c0e, log id: 71fdb627 >2012-12-17 10:06:55,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:06:55,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:55,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: d47d501 >2012-12-17 10:06:55,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: d47d501 >2012-12-17 10:06:55,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51ff7820 >2012-12-17 10:06:55,755 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:55,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6776f4c6 >2012-12-17 10:06:55,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:06:55,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1cab66a7 >2012-12-17 10:06:55,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1cab66a7 >2012-12-17 10:06:55,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-65) [302432ce] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6776f4c6 >2012-12-17 10:06:55,775 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-65) [302432ce] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:06:58,463 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:07:04,643 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6bb29080] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:07:04,778 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [6001c862] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:07:04,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e2cc9a4 >2012-12-17 10:07:04,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6aa77541 >2012-12-17 10:07:04,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:05,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, SpmStopVDSCommand, log id: 6aa77541 >2012-12-17 10:07:05,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, ResetIrsVDSCommand, log id: 2e2cc9a4 >2012-12-17 10:07:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30162110 >2012-12-17 10:07:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30162110 >2012-12-17 10:07:08,694 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:07:15,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:07:15,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5056 >2012-12-17 10:07:15,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5056, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b42a2b >2012-12-17 10:07:15,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling started: taskId = 9b32d14a-2a4a-4e02-a8a1-2a4433e2e543 >2012-12-17 10:07:16,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling ended: taskId = 9b32d14a-2a4a-4e02-a8a1-2a4433e2e543 task status = finished >2012-12-17 10:07:16,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] spmStart polling ended, spm status: SPM >2012-12-17 10:07:16,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [511011e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b32d14a-2a4a-4e02-a8a1-2a4433e2e543), log id: 715ee6ce >2012-12-17 10:07:16,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, HSMClearTaskVDSCommand, log id: 715ee6ce >2012-12-17 10:07:16,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [511011e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@d831888, log id: 3b42a2b >2012-12-17 10:07:16,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:07:16,638 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:07:16,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68e34d84 >2012-12-17 10:07:16,678 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [511011e7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:07:16,896 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [62b3971a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:07:16,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 281576f3 >2012-12-17 10:07:16,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7e88b361 >2012-12-17 10:07:16,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:17,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, SpmStopVDSCommand, log id: 7e88b361 >2012-12-17 10:07:17,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, ResetIrsVDSCommand, log id: 281576f3 >2012-12-17 10:07:17,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ed9a399 >2012-12-17 10:07:17,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [62b3971a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:17,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [62b3971a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ed9a399 >2012-12-17 10:07:17,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:07:17,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5057 >2012-12-17 10:07:17,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5057, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5869d64a >2012-12-17 10:07:17,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling started: taskId = 0c714cec-42f4-473a-8860-6dc2a43efae6 >2012-12-17 10:07:18,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling ended: taskId = 0c714cec-42f4-473a-8860-6dc2a43efae6 task status = finished >2012-12-17 10:07:18,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] spmStart polling ended, spm status: SPM >2012-12-17 10:07:18,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0c714cec-42f4-473a-8860-6dc2a43efae6), log id: 35f90494 >2012-12-17 10:07:18,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMClearTaskVDSCommand, log id: 35f90494 >2012-12-17 10:07:18,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f9bac3a, log id: 5869d64a >2012-12-17 10:07:18,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [56592212] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:07:18,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:18,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 29a377c4 >2012-12-17 10:07:18,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 29a377c4 >2012-12-17 10:07:18,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68e34d84 >2012-12-17 10:07:18,432 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:07:18,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1f5846f5 >2012-12-17 10:07:18,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:18,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25ff0ea4 >2012-12-17 10:07:18,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25ff0ea4 >2012-12-17 10:07:18,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-87) [56592212] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1f5846f5 >2012-12-17 10:07:18,451 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-87) [56592212] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:07:18,961 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:07:27,309 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2c358b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:07:27,403 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [1db2b76] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:07:27,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4958edb6 >2012-12-17 10:07:27,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 123a13b9 >2012-12-17 10:07:27,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:27,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, SpmStopVDSCommand, log id: 123a13b9 >2012-12-17 10:07:27,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, ResetIrsVDSCommand, log id: 4958edb6 >2012-12-17 10:07:27,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2beb2e21 >2012-12-17 10:07:27,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:27,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2beb2e21 >2012-12-17 10:07:29,183 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [5117cf2d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:07:37,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:07:37,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5058 >2012-12-17 10:07:37,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5058, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 406c7b93 >2012-12-17 10:07:37,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling started: taskId = 5789224d-ce39-43a6-bad4-d9ef62a04802 >2012-12-17 10:07:38,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling ended: taskId = 5789224d-ce39-43a6-bad4-d9ef62a04802 task status = finished >2012-12-17 10:07:38,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] spmStart polling ended, spm status: SPM >2012-12-17 10:07:38,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5789224d-ce39-43a6-bad4-d9ef62a04802), log id: ea3f1b5 >2012-12-17 10:07:38,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, HSMClearTaskVDSCommand, log id: ea3f1b5 >2012-12-17 10:07:38,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [2669bad8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d51a995, log id: 406c7b93 >2012-12-17 10:07:39,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:07:39,118 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:07:39,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7715af99 >2012-12-17 10:07:39,135 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2669bad8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:07:39,512 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [6018e3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:07:39,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17d73da9 >2012-12-17 10:07:39,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5dd8725b >2012-12-17 10:07:39,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:39,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, SpmStopVDSCommand, log id: 5dd8725b >2012-12-17 10:07:39,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, ResetIrsVDSCommand, log id: 17d73da9 >2012-12-17 10:07:39,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [6018e3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 651071a1 >2012-12-17 10:07:39,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [6018e3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:39,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [6018e3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 651071a1 >2012-12-17 10:07:39,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:07:39,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5059 >2012-12-17 10:07:39,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5059, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1dd9b5e4 >2012-12-17 10:07:39,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling started: taskId = ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c >2012-12-17 10:07:40,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling ended: taskId = ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c task status = finished >2012-12-17 10:07:40,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] spmStart polling ended, spm status: SPM >2012-12-17 10:07:40,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ad6bdd7a-4d68-4dd0-b83c-e7c63ad1ee6c), log id: 12bc8f78 >2012-12-17 10:07:40,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMClearTaskVDSCommand, log id: 12bc8f78 >2012-12-17 10:07:40,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@196f18e4, log id: 1dd9b5e4 >2012-12-17 10:07:41,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:07:41,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:41,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 135cf34c >2012-12-17 10:07:41,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 135cf34c >2012-12-17 10:07:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7715af99 >2012-12-17 10:07:41,793 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:07:41,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3b690c82 >2012-12-17 10:07:41,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:41,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6ec6a2c4 >2012-12-17 10:07:41,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6ec6a2c4 >2012-12-17 10:07:41,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-30) [5f725bc5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b690c82 >2012-12-17 10:07:41,813 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-30) [5f725bc5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:07:49,589 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:07:49,890 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7790038] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:07:49,999 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [6038c0b1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:07:50,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 672fe7e3 >2012-12-17 10:07:50,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 42ce78af >2012-12-17 10:07:50,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:50,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, SpmStopVDSCommand, log id: 42ce78af >2012-12-17 10:07:50,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, ResetIrsVDSCommand, log id: 672fe7e3 >2012-12-17 10:07:50,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43a8d6a9 >2012-12-17 10:07:50,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [6038c0b1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:07:50,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [6038c0b1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43a8d6a9 >2012-12-17 10:07:59,774 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:00,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:00,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5060 >2012-12-17 10:08:00,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5060, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1eeb9318 >2012-12-17 10:08:00,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling started: taskId = caf69730-de32-4f15-94f7-346277a49cb5 >2012-12-17 10:08:01,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling ended: taskId = caf69730-de32-4f15-94f7-346277a49cb5 task status = finished >2012-12-17 10:08:01,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] spmStart polling ended, spm status: SPM >2012-12-17 10:08:01,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [71a21411] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=caf69730-de32-4f15-94f7-346277a49cb5), log id: 65624a0c >2012-12-17 10:08:01,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, HSMClearTaskVDSCommand, log id: 65624a0c >2012-12-17 10:08:01,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [71a21411] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17eb8dd4, log id: 1eeb9318 >2012-12-17 10:08:01,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:01,874 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:08:01,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 203403ef >2012-12-17 10:08:01,902 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [71a21411] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:02,069 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:02,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 22d47806 >2012-12-17 10:08:02,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 23f7386f >2012-12-17 10:08:02,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:02,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, SpmStopVDSCommand, log id: 23f7386f >2012-12-17 10:08:02,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, ResetIrsVDSCommand, log id: 22d47806 >2012-12-17 10:08:02,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c5f1e77 >2012-12-17 10:08:02,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:02,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c5f1e77 >2012-12-17 10:08:02,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:02,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5061 >2012-12-17 10:08:02,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5061, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2638ac5e >2012-12-17 10:08:02,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling started: taskId = b2c233c1-bec2-4eca-b43c-3bce38107eb9 >2012-12-17 10:08:03,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended: taskId = b2c233c1-bec2-4eca-b43c-3bce38107eb9 task status = finished >2012-12-17 10:08:03,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] spmStart polling ended, spm status: SPM >2012-12-17 10:08:03,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b2c233c1-bec2-4eca-b43c-3bce38107eb9), log id: 6b5d619c >2012-12-17 10:08:03,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMClearTaskVDSCommand, log id: 6b5d619c >2012-12-17 10:08:03,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3fb37fd5, log id: 2638ac5e >2012-12-17 10:08:03,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [21122b75] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:03,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:03,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6cb8ca1a >2012-12-17 10:08:03,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6cb8ca1a >2012-12-17 10:08:03,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 203403ef >2012-12-17 10:08:03,604 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:03,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 57c936bf >2012-12-17 10:08:03,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:03,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2d8422ac >2012-12-17 10:08:03,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d8422ac >2012-12-17 10:08:03,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-45) [21122b75] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 57c936bf >2012-12-17 10:08:03,624 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-45) [21122b75] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:09,987 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:12,530 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [5920de03] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:12,827 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [40858b1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:12,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77c2d1bb >2012-12-17 10:08:12,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6f681fcd >2012-12-17 10:08:12,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:13,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, SpmStopVDSCommand, log id: 6f681fcd >2012-12-17 10:08:13,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, ResetIrsVDSCommand, log id: 77c2d1bb >2012-12-17 10:08:13,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78bc27f9 >2012-12-17 10:08:13,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:13,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78bc27f9 >2012-12-17 10:08:20,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1ecb2997] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:23,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:23,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5062 >2012-12-17 10:08:23,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5062, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65fa5608 >2012-12-17 10:08:23,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling started: taskId = 602db02f-92c1-43a0-982a-774f828f7159 >2012-12-17 10:08:24,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling ended: taskId = 602db02f-92c1-43a0-982a-774f828f7159 task status = finished >2012-12-17 10:08:24,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] spmStart polling ended, spm status: SPM >2012-12-17 10:08:24,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=602db02f-92c1-43a0-982a-774f828f7159), log id: 59f7e2fc >2012-12-17 10:08:24,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, HSMClearTaskVDSCommand, log id: 59f7e2fc >2012-12-17 10:08:24,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [8b964cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21573c89, log id: 65fa5608 >2012-12-17 10:08:24,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:24,722 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:08:24,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 33a4f9e2 >2012-12-17 10:08:24,754 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [8b964cc] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:24,861 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [a355c3f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:24,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5032e9d0 >2012-12-17 10:08:24,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f51391c >2012-12-17 10:08:24,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:25,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, SpmStopVDSCommand, log id: 4f51391c >2012-12-17 10:08:25,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, ResetIrsVDSCommand, log id: 5032e9d0 >2012-12-17 10:08:25,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 110294cc >2012-12-17 10:08:25,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:25,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 110294cc >2012-12-17 10:08:25,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:25,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5063 >2012-12-17 10:08:25,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5063, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 145e4a8 >2012-12-17 10:08:25,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = 06577d4a-8a7b-4684-bba0-8baf007bb20b >2012-12-17 10:08:26,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = 06577d4a-8a7b-4684-bba0-8baf007bb20b task status = finished >2012-12-17 10:08:26,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM >2012-12-17 10:08:26,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=06577d4a-8a7b-4684-bba0-8baf007bb20b), log id: 4a813a1e >2012-12-17 10:08:26,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 4a813a1e >2012-12-17 10:08:26,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4abde2be, log id: 145e4a8 >2012-12-17 10:08:26,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:26,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:26,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5028395f >2012-12-17 10:08:26,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5028395f >2012-12-17 10:08:26,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 33a4f9e2 >2012-12-17 10:08:26,836 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:26,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: ceb516 >2012-12-17 10:08:26,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:26,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4e6377de >2012-12-17 10:08:26,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e6377de >2012-12-17 10:08:26,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: ceb516 >2012-12-17 10:08:26,856 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-71) [34a366b8] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:30,398 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:35,201 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [20d09da7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:35,335 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [1290f5b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:35,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: cdcbd9 >2012-12-17 10:08:35,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 63a11b24 >2012-12-17 10:08:35,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:36,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, SpmStopVDSCommand, log id: 63a11b24 >2012-12-17 10:08:36,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, ResetIrsVDSCommand, log id: cdcbd9 >2012-12-17 10:08:36,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54552694 >2012-12-17 10:08:36,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [1290f5b5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:36,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [1290f5b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54552694 >2012-12-17 10:08:40,587 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:46,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:46,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5064 >2012-12-17 10:08:46,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5064, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e5070ea >2012-12-17 10:08:46,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling started: taskId = 4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84 >2012-12-17 10:08:47,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling ended: taskId = 4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84 task status = finished >2012-12-17 10:08:47,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] spmStart polling ended, spm status: SPM >2012-12-17 10:08:47,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [470bd088] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4b0a00c6-d3a6-47c0-8ba1-df96ccc72c84), log id: 40b3bf18 >2012-12-17 10:08:47,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, HSMClearTaskVDSCommand, log id: 40b3bf18 >2012-12-17 10:08:47,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [470bd088] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c6e1d7, log id: 6e5070ea >2012-12-17 10:08:47,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:47,933 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:08:47,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7242ec4 >2012-12-17 10:08:47,953 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [470bd088] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:48,096 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [fc992c9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:48,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5214f5ea >2012-12-17 10:08:48,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 673e5de2 >2012-12-17 10:08:48,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:48,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, SpmStopVDSCommand, log id: 673e5de2 >2012-12-17 10:08:48,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, ResetIrsVDSCommand, log id: 5214f5ea >2012-12-17 10:08:48,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c3f6653 >2012-12-17 10:08:48,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:48,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c3f6653 >2012-12-17 10:08:48,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:08:48,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5065 >2012-12-17 10:08:48,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5065, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a06a41b >2012-12-17 10:08:48,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = 4bb6f221-dec2-4688-a27a-5d1f40b61a71 >2012-12-17 10:08:49,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = 4bb6f221-dec2-4688-a27a-5d1f40b61a71 task status = finished >2012-12-17 10:08:49,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM >2012-12-17 10:08:49,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4bb6f221-dec2-4688-a27a-5d1f40b61a71), log id: 348837e3 >2012-12-17 10:08:49,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 348837e3 >2012-12-17 10:08:49,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b0d3306, log id: a06a41b >2012-12-17 10:08:49,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:08:49,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:49,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2fff4424 >2012-12-17 10:08:49,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2fff4424 >2012-12-17 10:08:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7242ec4 >2012-12-17 10:08:49,546 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1fd2b01a >2012-12-17 10:08:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:49,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4623831a >2012-12-17 10:08:49,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4623831a >2012-12-17 10:08:49,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1fd2b01a >2012-12-17 10:08:49,566 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:08:50,766 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:08:58,440 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [786860d5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:08:58,585 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [66f997e3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:08:58,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 646c28d4 >2012-12-17 10:08:58,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4a397745 >2012-12-17 10:08:58,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:58,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, SpmStopVDSCommand, log id: 4a397745 >2012-12-17 10:08:58,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, ResetIrsVDSCommand, log id: 646c28d4 >2012-12-17 10:08:58,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15cd5966 >2012-12-17 10:08:58,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [66f997e3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:08:58,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [66f997e3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15cd5966 >2012-12-17 10:09:01,843 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:09:09,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:09,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5066 >2012-12-17 10:09:09,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5066, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d4c679f >2012-12-17 10:09:09,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling started: taskId = 2f0b7400-64b9-4c87-91a6-5a7863855b16 >2012-12-17 10:09:11,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling ended: taskId = 2f0b7400-64b9-4c87-91a6-5a7863855b16 task status = finished >2012-12-17 10:09:11,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] spmStart polling ended, spm status: SPM >2012-12-17 10:09:11,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2f0b7400-64b9-4c87-91a6-5a7863855b16), log id: 5a0b2d54 >2012-12-17 10:09:11,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, HSMClearTaskVDSCommand, log id: 5a0b2d54 >2012-12-17 10:09:11,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13cd13fb, log id: 7d4c679f >2012-12-17 10:09:11,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:09:11,465 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:09:11,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 392366fb >2012-12-17 10:09:11,487 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [1e7a0ae1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:09:11,601 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [55bb30e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:09:11,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 586a50a6 >2012-12-17 10:09:11,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3afc5b6f >2012-12-17 10:09:11,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:11,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SpmStopVDSCommand, log id: 3afc5b6f >2012-12-17 10:09:11,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, ResetIrsVDSCommand, log id: 586a50a6 >2012-12-17 10:09:11,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2536eab8 >2012-12-17 10:09:11,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:11,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2536eab8 >2012-12-17 10:09:11,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:12,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5067 >2012-12-17 10:09:12,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5067, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34c30eae >2012-12-17 10:09:12,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling started: taskId = 0b8fa183-02a7-46fd-8a77-df06fde89d1a >2012-12-17 10:09:13,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling ended: taskId = 0b8fa183-02a7-46fd-8a77-df06fde89d1a task status = finished >2012-12-17 10:09:13,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] spmStart polling ended, spm status: SPM >2012-12-17 10:09:13,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0b8fa183-02a7-46fd-8a77-df06fde89d1a), log id: 4b7be55c >2012-12-17 10:09:13,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMClearTaskVDSCommand, log id: 4b7be55c >2012-12-17 10:09:13,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27076335, log id: 34c30eae >2012-12-17 10:09:13,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:09:13,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:13,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5cd2f6eb >2012-12-17 10:09:13,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5cd2f6eb >2012-12-17 10:09:13,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 392366fb >2012-12-17 10:09:13,171 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:09:13,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cea2427 >2012-12-17 10:09:13,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:13,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 23f5ef78 >2012-12-17 10:09:13,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23f5ef78 >2012-12-17 10:09:13,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-78) [569d76f1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cea2427 >2012-12-17 10:09:13,192 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-78) [569d76f1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:09:22,011 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [6827474] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:09:22,112 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [56306d9f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:09:22,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3f0fc259 >2012-12-17 10:09:22,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7b47292 >2012-12-17 10:09:22,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:22,324 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:09:22,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, SpmStopVDSCommand, log id: 7b47292 >2012-12-17 10:09:23,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, ResetIrsVDSCommand, log id: 3f0fc259 >2012-12-17 10:09:23,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ee89afe >2012-12-17 10:09:23,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:23,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ee89afe >2012-12-17 10:09:33,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:33,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5068 >2012-12-17 10:09:33,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5068, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73a5c1dd >2012-12-17 10:09:33,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling started: taskId = 88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e >2012-12-17 10:09:34,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling ended: taskId = 88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e task status = finished >2012-12-17 10:09:34,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] spmStart polling ended, spm status: SPM >2012-12-17 10:09:34,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=88c02bdf-60f4-43ee-b58c-a0d5e8bcf47e), log id: 73a88f38 >2012-12-17 10:09:34,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, HSMClearTaskVDSCommand, log id: 73a88f38 >2012-12-17 10:09:34,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [4d407d58] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f52e0da, log id: 73a5c1dd >2012-12-17 10:09:34,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:09:34,240 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:09:34,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 53044205 >2012-12-17 10:09:34,262 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4d407d58] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:09:34,360 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [7aeab1da] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:09:34,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 718b2191 >2012-12-17 10:09:34,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d8a3737 >2012-12-17 10:09:34,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:35,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, SpmStopVDSCommand, log id: 6d8a3737 >2012-12-17 10:09:35,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, ResetIrsVDSCommand, log id: 718b2191 >2012-12-17 10:09:35,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3297a447 >2012-12-17 10:09:35,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:35,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3297a447 >2012-12-17 10:09:35,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:35,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5069 >2012-12-17 10:09:35,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5069, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bbd9522 >2012-12-17 10:09:35,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling started: taskId = e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b >2012-12-17 10:09:36,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling ended: taskId = e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b task status = finished >2012-12-17 10:09:36,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] spmStart polling ended, spm status: SPM >2012-12-17 10:09:36,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e7233cc1-49b9-4d5f-825c-39b7f0ee0c5b), log id: 3e99c483 >2012-12-17 10:09:36,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMClearTaskVDSCommand, log id: 3e99c483 >2012-12-17 10:09:36,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c4d81e4, log id: 6bbd9522 >2012-12-17 10:09:36,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:09:36,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:36,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75f550ca >2012-12-17 10:09:36,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75f550ca >2012-12-17 10:09:36,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 53044205 >2012-12-17 10:09:36,809 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:09:36,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c8b3a46 >2012-12-17 10:09:36,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:36,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f4ad571 >2012-12-17 10:09:36,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f4ad571 >2012-12-17 10:09:36,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-38) [7dff3777] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3c8b3a46 >2012-12-17 10:09:36,830 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-38) [7dff3777] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:09:42,764 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:09:45,361 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [140420a5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:09:45,566 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [b18376] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:09:45,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5b706106 >2012-12-17 10:09:45,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24946378 >2012-12-17 10:09:45,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:45,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, SpmStopVDSCommand, log id: 24946378 >2012-12-17 10:09:46,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, ResetIrsVDSCommand, log id: 5b706106 >2012-12-17 10:09:46,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [b18376] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21df4b6e >2012-12-17 10:09:46,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [b18376] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:46,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [b18376] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21df4b6e >2012-12-17 10:09:52,974 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:09:56,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:56,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5070 >2012-12-17 10:09:56,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5070, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76092f7c >2012-12-17 10:09:56,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling started: taskId = 4c9385b6-0830-43da-ba0b-4483c298dbd5 >2012-12-17 10:09:57,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling ended: taskId = 4c9385b6-0830-43da-ba0b-4483c298dbd5 task status = finished >2012-12-17 10:09:57,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] spmStart polling ended, spm status: SPM >2012-12-17 10:09:57,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4c9385b6-0830-43da-ba0b-4483c298dbd5), log id: 3717367e >2012-12-17 10:09:57,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, HSMClearTaskVDSCommand, log id: 3717367e >2012-12-17 10:09:57,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) [4a8b829d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54faa673, log id: 76092f7c >2012-12-17 10:09:57,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:09:57,994 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:09:57,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2496725f >2012-12-17 10:09:58,011 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [4a8b829d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:09:58,143 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:09:58,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e9723c >2012-12-17 10:09:58,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4fb38fc8 >2012-12-17 10:09:58,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:58,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, SpmStopVDSCommand, log id: 4fb38fc8 >2012-12-17 10:09:58,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, ResetIrsVDSCommand, log id: 2e9723c >2012-12-17 10:09:58,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44a5a8a8 >2012-12-17 10:09:58,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:09:58,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [3b6c2c0a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44a5a8a8 >2012-12-17 10:09:59,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:09:59,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5071 >2012-12-17 10:09:59,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5071, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44e22257 >2012-12-17 10:09:59,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling started: taskId = 42a9731d-efa7-45f3-b893-ae80fa8971b1 >2012-12-17 10:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Checking autorecoverable hosts >2012-12-17 10:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Autorecovering 0 hosts >2012-12-17 10:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Checking autorecoverable hosts done >2012-12-17 10:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [29a5a9b6] Autorecovering storage domains is disabled, skipping >2012-12-17 10:10:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended: taskId = 42a9731d-efa7-45f3-b893-ae80fa8971b1 task status = finished >2012-12-17 10:10:00,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended, spm status: SPM >2012-12-17 10:10:00,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=42a9731d-efa7-45f3-b893-ae80fa8971b1), log id: 47e964c0 >2012-12-17 10:10:00,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMClearTaskVDSCommand, log id: 47e964c0 >2012-12-17 10:10:00,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2bd59e22, log id: 44e22257 >2012-12-17 10:10:00,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:10:00,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:00,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3f0eab15 >2012-12-17 10:10:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3f0eab15 >2012-12-17 10:10:00,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2496725f >2012-12-17 10:10:00,164 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:00,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b6b2fb7 >2012-12-17 10:10:00,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 57292bf2 >2012-12-17 10:10:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 57292bf2 >2012-12-17 10:10:00,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: b6b2fb7 >2012-12-17 10:10:00,184 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:03,220 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [5bc6d47] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:10:09,052 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [56dc6659] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:10:09,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [5b09131c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:10:09,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7a91f1b >2012-12-17 10:10:09,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 30ecb2eb >2012-12-17 10:10:09,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:09,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, SpmStopVDSCommand, log id: 30ecb2eb >2012-12-17 10:10:09,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, ResetIrsVDSCommand, log id: 7a91f1b >2012-12-17 10:10:09,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ae6d2cc >2012-12-17 10:10:09,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [5b09131c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:09,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [5b09131c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ae6d2cc >2012-12-17 10:10:13,420 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:10:19,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:10:19,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5072 >2012-12-17 10:10:19,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5072, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45d62007 >2012-12-17 10:10:19,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling started: taskId = 77ab263c-03aa-43e0-8171-7bd24b17d10a >2012-12-17 10:10:20,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling ended: taskId = 77ab263c-03aa-43e0-8171-7bd24b17d10a task status = finished >2012-12-17 10:10:20,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] spmStart polling ended, spm status: SPM >2012-12-17 10:10:20,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=77ab263c-03aa-43e0-8171-7bd24b17d10a), log id: 1ed1c0c >2012-12-17 10:10:20,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, HSMClearTaskVDSCommand, log id: 1ed1c0c >2012-12-17 10:10:20,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [464dfbdd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75cc58d0, log id: 45d62007 >2012-12-17 10:10:21,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:10:22,106 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:10:22,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e87fa84 >2012-12-17 10:10:22,107 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [464dfbdd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:10:22,577 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [141038b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:10:22,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: b0eaaa3 >2012-12-17 10:10:22,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7701ea41 >2012-12-17 10:10:22,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:23,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SpmStopVDSCommand, log id: 7701ea41 >2012-12-17 10:10:23,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, ResetIrsVDSCommand, log id: b0eaaa3 >2012-12-17 10:10:23,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5488eca9 >2012-12-17 10:10:23,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:23,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5488eca9 >2012-12-17 10:10:23,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:10:23,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5073 >2012-12-17 10:10:23,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5073, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15e092bd >2012-12-17 10:10:23,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 3f4f06bc-0c36-4efd-8896-0b799e54dfe4 >2012-12-17 10:10:24,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 3f4f06bc-0c36-4efd-8896-0b799e54dfe4 task status = finished >2012-12-17 10:10:24,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM >2012-12-17 10:10:24,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3f4f06bc-0c36-4efd-8896-0b799e54dfe4), log id: 3aeb3bbf >2012-12-17 10:10:24,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 3aeb3bbf >2012-12-17 10:10:24,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6312ff79, log id: 15e092bd >2012-12-17 10:10:24,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:10:24,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:24,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d6a7cfb >2012-12-17 10:10:24,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d6a7cfb >2012-12-17 10:10:24,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e87fa84 >2012-12-17 10:10:24,461 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:24,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 35916799 >2012-12-17 10:10:24,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:24,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6121e691 >2012-12-17 10:10:24,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6121e691 >2012-12-17 10:10:24,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 35916799 >2012-12-17 10:10:24,482 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-12) [1407e3a2] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:33,308 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [375e73d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:10:33,400 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [15d070bf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:10:33,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 51d5f396 >2012-12-17 10:10:33,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 50033860 >2012-12-17 10:10:33,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:33,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SpmStopVDSCommand, log id: 50033860 >2012-12-17 10:10:33,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, ResetIrsVDSCommand, log id: 51d5f396 >2012-12-17 10:10:33,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5cac45bc >2012-12-17 10:10:33,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:33,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5cac45bc >2012-12-17 10:10:33,812 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [3d52876a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:10:43,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:10:43,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5074 >2012-12-17 10:10:43,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5074, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13b98396 >2012-12-17 10:10:43,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling started: taskId = 3753fb83-295b-4b4e-8f4a-f3bfe5146769 >2012-12-17 10:10:44,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling ended: taskId = 3753fb83-295b-4b4e-8f4a-f3bfe5146769 task status = finished >2012-12-17 10:10:44,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] spmStart polling ended, spm status: SPM >2012-12-17 10:10:44,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [6412070c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3753fb83-295b-4b4e-8f4a-f3bfe5146769), log id: 3cc01920 >2012-12-17 10:10:44,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, HSMClearTaskVDSCommand, log id: 3cc01920 >2012-12-17 10:10:44,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [6412070c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67b1a308, log id: 13b98396 >2012-12-17 10:10:44,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:10:45,015 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:10:45,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b424c05 >2012-12-17 10:10:45,031 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [6412070c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:10:45,158 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [102f58ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:10:45,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49aa8082 >2012-12-17 10:10:45,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 9d05794 >2012-12-17 10:10:45,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:45,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, SpmStopVDSCommand, log id: 9d05794 >2012-12-17 10:10:45,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, ResetIrsVDSCommand, log id: 49aa8082 >2012-12-17 10:10:45,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 38c5d7f6 >2012-12-17 10:10:45,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:45,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [102f58ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 38c5d7f6 >2012-12-17 10:10:45,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:10:45,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5075 >2012-12-17 10:10:45,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5075, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fc80f20 >2012-12-17 10:10:45,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling started: taskId = 46bda918-1719-43ff-949e-76e1342f143b >2012-12-17 10:10:46,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended: taskId = 46bda918-1719-43ff-949e-76e1342f143b task status = finished >2012-12-17 10:10:46,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] spmStart polling ended, spm status: SPM >2012-12-17 10:10:46,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=46bda918-1719-43ff-949e-76e1342f143b), log id: 1a510a88 >2012-12-17 10:10:46,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMClearTaskVDSCommand, log id: 1a510a88 >2012-12-17 10:10:46,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e8a7437, log id: 2fc80f20 >2012-12-17 10:10:46,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:10:46,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:46,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 25fb17bd >2012-12-17 10:10:46,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25fb17bd >2012-12-17 10:10:46,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b424c05 >2012-12-17 10:10:46,664 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:46,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51bf1f2d >2012-12-17 10:10:46,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 56d0dca8 >2012-12-17 10:10:46,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56d0dca8 >2012-12-17 10:10:46,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [5654bc9d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 51bf1f2d >2012-12-17 10:10:46,684 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [5654bc9d] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:10:54,378 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:10:55,506 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [2f07235e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:10:55,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [78bc48c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:10:55,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4f50de86 >2012-12-17 10:10:55,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 452de4b9 >2012-12-17 10:10:55,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:56,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, SpmStopVDSCommand, log id: 452de4b9 >2012-12-17 10:10:56,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, ResetIrsVDSCommand, log id: 4f50de86 >2012-12-17 10:10:56,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fd91ee8 >2012-12-17 10:10:56,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:10:56,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fd91ee8 >2012-12-17 10:11:04,696 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:11:06,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:07,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5076 >2012-12-17 10:11:07,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5076, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65848726 >2012-12-17 10:11:07,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling started: taskId = 8b61ee91-75b7-489a-a6e0-205e40c785ef >2012-12-17 10:11:08,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling ended: taskId = 8b61ee91-75b7-489a-a6e0-205e40c785ef task status = finished >2012-12-17 10:11:08,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] spmStart polling ended, spm status: SPM >2012-12-17 10:11:08,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8b61ee91-75b7-489a-a6e0-205e40c785ef), log id: 7f2425b3 >2012-12-17 10:11:08,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, HSMClearTaskVDSCommand, log id: 7f2425b3 >2012-12-17 10:11:08,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [37e97c0a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e274bfb, log id: 65848726 >2012-12-17 10:11:08,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:08,164 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:11:08,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 48730772 >2012-12-17 10:11:08,174 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [37e97c0a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:11:08,329 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [68ede0fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:11:08,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 13ecf4a2 >2012-12-17 10:11:08,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5199f93d >2012-12-17 10:11:08,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:08,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SpmStopVDSCommand, log id: 5199f93d >2012-12-17 10:11:08,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, ResetIrsVDSCommand, log id: 13ecf4a2 >2012-12-17 10:11:08,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66941b8d >2012-12-17 10:11:08,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:08,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66941b8d >2012-12-17 10:11:08,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:08,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5077 >2012-12-17 10:11:08,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5077, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 67bb3496 >2012-12-17 10:11:08,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling started: taskId = 6dc969dd-f65f-46dd-9ddb-3aab3c48a907 >2012-12-17 10:11:09,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended: taskId = 6dc969dd-f65f-46dd-9ddb-3aab3c48a907 task status = finished >2012-12-17 10:11:09,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] spmStart polling ended, spm status: SPM >2012-12-17 10:11:09,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6dc969dd-f65f-46dd-9ddb-3aab3c48a907), log id: 21d07285 >2012-12-17 10:11:09,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMClearTaskVDSCommand, log id: 21d07285 >2012-12-17 10:11:09,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c0e8b1a, log id: 67bb3496 >2012-12-17 10:11:09,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2d67b14a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:09,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:09,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4318a70 >2012-12-17 10:11:09,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4318a70 >2012-12-17 10:11:09,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 48730772 >2012-12-17 10:11:09,849 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:09,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 662de97f >2012-12-17 10:11:09,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:09,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 600e1518 >2012-12-17 10:11:09,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 600e1518 >2012-12-17 10:11:09,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-72) [2d67b14a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 662de97f >2012-12-17 10:11:09,869 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) [2d67b14a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:14,918 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:11:18,871 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [5cd90aa7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:11:19,013 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [1dd89be6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:11:19,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 14f2ceb4 >2012-12-17 10:11:19,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e850107 >2012-12-17 10:11:19,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:19,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SpmStopVDSCommand, log id: 2e850107 >2012-12-17 10:11:19,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, ResetIrsVDSCommand, log id: 14f2ceb4 >2012-12-17 10:11:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1020ace1 >2012-12-17 10:11:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:19,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1020ace1 >2012-12-17 10:11:25,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:11:29,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:29,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5078 >2012-12-17 10:11:29,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5078, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4c1628b >2012-12-17 10:11:29,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling started: taskId = 38babc83-24be-4dba-a821-a8ee80d7f944 >2012-12-17 10:11:30,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling ended: taskId = 38babc83-24be-4dba-a821-a8ee80d7f944 task status = finished >2012-12-17 10:11:30,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] spmStart polling ended, spm status: SPM >2012-12-17 10:11:30,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38babc83-24be-4dba-a821-a8ee80d7f944), log id: 112244d1 >2012-12-17 10:11:30,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, HSMClearTaskVDSCommand, log id: 112244d1 >2012-12-17 10:11:30,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [2ce0ef25] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60d3fe20, log id: 4c1628b >2012-12-17 10:11:31,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:31,651 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:11:31,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39cba23e >2012-12-17 10:11:31,698 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [2ce0ef25] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:11:31,839 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [a97381c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:11:31,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1b8b7bcb >2012-12-17 10:11:31,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6d2d914a >2012-12-17 10:11:31,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:32,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, SpmStopVDSCommand, log id: 6d2d914a >2012-12-17 10:11:32,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, ResetIrsVDSCommand, log id: 1b8b7bcb >2012-12-17 10:11:32,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [a97381c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 159f30a6 >2012-12-17 10:11:32,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [a97381c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:32,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [a97381c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 159f30a6 >2012-12-17 10:11:32,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:32,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5079 >2012-12-17 10:11:32,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5079, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f9bae7e >2012-12-17 10:11:32,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling started: taskId = db7100c3-d74b-4fbe-818e-97094e5e55ef >2012-12-17 10:11:33,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling ended: taskId = db7100c3-d74b-4fbe-818e-97094e5e55ef task status = finished >2012-12-17 10:11:33,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] spmStart polling ended, spm status: SPM >2012-12-17 10:11:33,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=db7100c3-d74b-4fbe-818e-97094e5e55ef), log id: 7e34a0ee >2012-12-17 10:11:33,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMClearTaskVDSCommand, log id: 7e34a0ee >2012-12-17 10:11:33,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3131a9fd, log id: 6f9bae7e >2012-12-17 10:11:33,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:33,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:33,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 62da5df5 >2012-12-17 10:11:33,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 62da5df5 >2012-12-17 10:11:33,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39cba23e >2012-12-17 10:11:33,372 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:33,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e4eb286 >2012-12-17 10:11:33,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:33,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 20ccb3f8 >2012-12-17 10:11:33,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 20ccb3f8 >2012-12-17 10:11:33,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-48) [53480d94] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e4eb286 >2012-12-17 10:11:33,392 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-48) [53480d94] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:35,441 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:11:42,243 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [2a110a6a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:11:42,357 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [30ddd635] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:11:42,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3807089a >2012-12-17 10:11:42,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5ab26649 >2012-12-17 10:11:42,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:42,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SpmStopVDSCommand, log id: 5ab26649 >2012-12-17 10:11:42,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, ResetIrsVDSCommand, log id: 3807089a >2012-12-17 10:11:42,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 253639ab >2012-12-17 10:11:42,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:42,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 253639ab >2012-12-17 10:11:45,661 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:11:52,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:52,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5080 >2012-12-17 10:11:52,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5080, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a38e83e >2012-12-17 10:11:52,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling started: taskId = a80cd820-a6b4-48ad-92fc-17e8eb4abca7 >2012-12-17 10:11:53,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended: taskId = a80cd820-a6b4-48ad-92fc-17e8eb4abca7 task status = finished >2012-12-17 10:11:53,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] spmStart polling ended, spm status: SPM >2012-12-17 10:11:53,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a80cd820-a6b4-48ad-92fc-17e8eb4abca7), log id: 1d6f3ece >2012-12-17 10:11:53,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, HSMClearTaskVDSCommand, log id: 1d6f3ece >2012-12-17 10:11:53,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [75733504] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7284557a, log id: 4a38e83e >2012-12-17 10:11:53,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:53,921 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:11:53,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bc7af >2012-12-17 10:11:53,964 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [75733504] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:11:54,134 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [7e43b5c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:11:54,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 77467291 >2012-12-17 10:11:54,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 521cc783 >2012-12-17 10:11:54,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:54,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, SpmStopVDSCommand, log id: 521cc783 >2012-12-17 10:11:54,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, ResetIrsVDSCommand, log id: 77467291 >2012-12-17 10:11:54,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79b50160 >2012-12-17 10:11:54,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [7e43b5c5] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:54,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [7e43b5c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79b50160 >2012-12-17 10:11:54,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:11:54,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5081 >2012-12-17 10:11:54,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5081, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1509633a >2012-12-17 10:11:54,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling started: taskId = 3b453d1b-6459-4a81-862a-d8838f2a4bd7 >2012-12-17 10:11:55,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling ended: taskId = 3b453d1b-6459-4a81-862a-d8838f2a4bd7 task status = finished >2012-12-17 10:11:55,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] spmStart polling ended, spm status: SPM >2012-12-17 10:11:55,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3b453d1b-6459-4a81-862a-d8838f2a4bd7), log id: 33651cc2 >2012-12-17 10:11:55,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMClearTaskVDSCommand, log id: 33651cc2 >2012-12-17 10:11:55,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33431eea, log id: 1509633a >2012-12-17 10:11:55,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:11:55,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:55,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3e3d2976 >2012-12-17 10:11:55,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3e3d2976 >2012-12-17 10:11:55,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bc7af >2012-12-17 10:11:55,603 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:55,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3cc942f >2012-12-17 10:11:55,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:11:55,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4bf2bf37 >2012-12-17 10:11:55,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4bf2bf37 >2012-12-17 10:11:55,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-5) [66369950] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3cc942f >2012-12-17 10:11:55,622 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-5) [66369950] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:11:57,345 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:12:04,482 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [62b3971a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:12:04,586 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [76c4364c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:12:04,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: d5cb2d3 >2012-12-17 10:12:04,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 28926125 >2012-12-17 10:12:04,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:04,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, SpmStopVDSCommand, log id: 28926125 >2012-12-17 10:12:04,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, ResetIrsVDSCommand, log id: d5cb2d3 >2012-12-17 10:12:04,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bc2a644 >2012-12-17 10:12:04,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:04,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bc2a644 >2012-12-17 10:12:07,592 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:12:14,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:12:14,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5082 >2012-12-17 10:12:14,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5082, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7aa587c >2012-12-17 10:12:15,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling started: taskId = 0e9d6095-fd53-441e-9139-0c35ffe538e5 >2012-12-17 10:12:16,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling ended: taskId = 0e9d6095-fd53-441e-9139-0c35ffe538e5 task status = finished >2012-12-17 10:12:16,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] spmStart polling ended, spm status: SPM >2012-12-17 10:12:16,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=0e9d6095-fd53-441e-9139-0c35ffe538e5), log id: 43545336 >2012-12-17 10:12:16,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, HSMClearTaskVDSCommand, log id: 43545336 >2012-12-17 10:12:16,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) [65f4d22d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@552320c2, log id: 7aa587c >2012-12-17 10:12:16,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:12:16,974 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:12:16,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 37587f02 >2012-12-17 10:12:17,060 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [65f4d22d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:12:17,213 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [1630089b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:12:17,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 9edcd4d >2012-12-17 10:12:17,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 96832f >2012-12-17 10:12:17,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:17,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, SpmStopVDSCommand, log id: 96832f >2012-12-17 10:12:17,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, ResetIrsVDSCommand, log id: 9edcd4d >2012-12-17 10:12:17,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [1630089b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4384e88f >2012-12-17 10:12:17,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [1630089b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:17,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [1630089b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4384e88f >2012-12-17 10:12:17,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:12:17,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5083 >2012-12-17 10:12:17,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5083, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79459cb1 >2012-12-17 10:12:17,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling started: taskId = f2e51791-d2a4-4116-8fcd-c7493ea66fc2 >2012-12-17 10:12:18,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended: taskId = f2e51791-d2a4-4116-8fcd-c7493ea66fc2 task status = finished >2012-12-17 10:12:18,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended, spm status: SPM >2012-12-17 10:12:18,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f2e51791-d2a4-4116-8fcd-c7493ea66fc2), log id: 7418afb4 >2012-12-17 10:12:18,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMClearTaskVDSCommand, log id: 7418afb4 >2012-12-17 10:12:18,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@b78df8, log id: 79459cb1 >2012-12-17 10:12:18,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:12:18,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:18,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4996aeb4 >2012-12-17 10:12:18,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4996aeb4 >2012-12-17 10:12:18,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 37587f02 >2012-12-17 10:12:18,742 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:12:18,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e953f19 >2012-12-17 10:12:18,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:18,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 409e954c >2012-12-17 10:12:18,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 409e954c >2012-12-17 10:12:18,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3e953f19 >2012-12-17 10:12:18,762 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-54) [75689c2c] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:12:27,602 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5f725bc5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:12:27,809 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [2a51251c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:12:27,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6c03eee6 >2012-12-17 10:12:27,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 158d24c1 >2012-12-17 10:12:27,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:28,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, SpmStopVDSCommand, log id: 158d24c1 >2012-12-17 10:12:28,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, ResetIrsVDSCommand, log id: 6c03eee6 >2012-12-17 10:12:28,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a901274 >2012-12-17 10:12:28,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:28,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a901274 >2012-12-17 10:12:28,141 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [5074b866] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:12:38,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:12:38,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5084 >2012-12-17 10:12:38,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5084, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 346ad80a >2012-12-17 10:12:38,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling started: taskId = 76e5bba9-93bc-43eb-a2d5-d7860de506a5 >2012-12-17 10:12:39,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling ended: taskId = 76e5bba9-93bc-43eb-a2d5-d7860de506a5 task status = finished >2012-12-17 10:12:39,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] spmStart polling ended, spm status: SPM >2012-12-17 10:12:39,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [3577b816] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=76e5bba9-93bc-43eb-a2d5-d7860de506a5), log id: 77edd467 >2012-12-17 10:12:39,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, HSMClearTaskVDSCommand, log id: 77edd467 >2012-12-17 10:12:39,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [3577b816] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f13f3b4, log id: 346ad80a >2012-12-17 10:12:39,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:12:39,383 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:12:39,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e51a8b8 >2012-12-17 10:12:39,408 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [3577b816] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:12:39,678 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [956a2f9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:12:39,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5783844a >2012-12-17 10:12:39,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 224bd27f >2012-12-17 10:12:39,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:39,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, SpmStopVDSCommand, log id: 224bd27f >2012-12-17 10:12:39,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, ResetIrsVDSCommand, log id: 5783844a >2012-12-17 10:12:39,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1db423fb >2012-12-17 10:12:39,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [956a2f9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:39,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [956a2f9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1db423fb >2012-12-17 10:12:40,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:12:40,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5085 >2012-12-17 10:12:40,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5085, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 25dfaf06 >2012-12-17 10:12:40,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = a4ec3598-0730-4372-bc48-40686c304749 >2012-12-17 10:12:41,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = a4ec3598-0730-4372-bc48-40686c304749 task status = finished >2012-12-17 10:12:41,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM >2012-12-17 10:12:41,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a4ec3598-0730-4372-bc48-40686c304749), log id: 1c4dc490 >2012-12-17 10:12:41,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 1c4dc490 >2012-12-17 10:12:41,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35a3d874, log id: 25dfaf06 >2012-12-17 10:12:41,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:12:41,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:41,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43d6b33f >2012-12-17 10:12:41,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43d6b33f >2012-12-17 10:12:41,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e51a8b8 >2012-12-17 10:12:41,610 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:12:41,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 44920fe4 >2012-12-17 10:12:41,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:41,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 520031a6 >2012-12-17 10:12:41,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 520031a6 >2012-12-17 10:12:41,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 44920fe4 >2012-12-17 10:12:41,629 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-40) [346a0820] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:12:48,545 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:12:50,069 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [622cf65b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:12:50,194 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [3c648024] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:12:50,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 49183305 >2012-12-17 10:12:50,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4795323 >2012-12-17 10:12:50,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:50,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, SpmStopVDSCommand, log id: 4795323 >2012-12-17 10:12:50,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, ResetIrsVDSCommand, log id: 49183305 >2012-12-17 10:12:50,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22cde959 >2012-12-17 10:12:50,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:12:50,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22cde959 >2012-12-17 10:12:58,782 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:00,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:00,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5086 >2012-12-17 10:13:00,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5086, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c253d87 >2012-12-17 10:13:00,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling started: taskId = 1164a0e7-7564-46a1-bcc1-0450227fd116 >2012-12-17 10:13:01,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling ended: taskId = 1164a0e7-7564-46a1-bcc1-0450227fd116 task status = finished >2012-12-17 10:13:01,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] spmStart polling ended, spm status: SPM >2012-12-17 10:13:01,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1164a0e7-7564-46a1-bcc1-0450227fd116), log id: 6ccaea2c >2012-12-17 10:13:01,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, HSMClearTaskVDSCommand, log id: 6ccaea2c >2012-12-17 10:13:01,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [40858b1a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7dd42680, log id: 3c253d87 >2012-12-17 10:13:01,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:01,892 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:13:01,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8758276 >2012-12-17 10:13:01,929 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [40858b1a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:02,092 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [46423891] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:02,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 533f5f81 >2012-12-17 10:13:02,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6c16c8ee >2012-12-17 10:13:02,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, SpmStopVDSCommand, log id: 6c16c8ee >2012-12-17 10:13:02,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, ResetIrsVDSCommand, log id: 533f5f81 >2012-12-17 10:13:02,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [46423891] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 435e6064 >2012-12-17 10:13:02,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [46423891] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:02,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [46423891] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 435e6064 >2012-12-17 10:13:02,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:02,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5087 >2012-12-17 10:13:02,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5087, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ed55267 >2012-12-17 10:13:02,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling started: taskId = 9b634c12-e229-4665-abd4-a821b421fbc0 >2012-12-17 10:13:03,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended: taskId = 9b634c12-e229-4665-abd4-a821b421fbc0 task status = finished >2012-12-17 10:13:03,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] spmStart polling ended, spm status: SPM >2012-12-17 10:13:03,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=9b634c12-e229-4665-abd4-a821b421fbc0), log id: 2b2656af >2012-12-17 10:13:03,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMClearTaskVDSCommand, log id: 2b2656af >2012-12-17 10:13:03,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6205114a, log id: ed55267 >2012-12-17 10:13:03,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:03,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:03,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3ff8e0e5 >2012-12-17 10:13:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3ff8e0e5 >2012-12-17 10:13:03,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 8758276 >2012-12-17 10:13:03,687 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:03,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c74fb32 >2012-12-17 10:13:03,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:03,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4c3838df >2012-12-17 10:13:03,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4c3838df >2012-12-17 10:13:03,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6c74fb32 >2012-12-17 10:13:03,707 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) [4d8dc1c4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:09,089 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [53480d94] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:12,529 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [4f3078ed] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:12,935 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [10ed878d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:13,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44b250ed >2012-12-17 10:13:13,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 64d1baa2 >2012-12-17 10:13:13,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:13,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, SpmStopVDSCommand, log id: 64d1baa2 >2012-12-17 10:13:13,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, ResetIrsVDSCommand, log id: 44b250ed >2012-12-17 10:13:13,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 501f6383 >2012-12-17 10:13:13,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [10ed878d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:13,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [10ed878d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 501f6383 >2012-12-17 10:13:19,333 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:23,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:23,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5088 >2012-12-17 10:13:23,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5088, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d3573df >2012-12-17 10:13:23,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling started: taskId = e990e37b-fe00-4b7c-a447-7a3953c466d5 >2012-12-17 10:13:24,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling ended: taskId = e990e37b-fe00-4b7c-a447-7a3953c466d5 task status = finished >2012-12-17 10:13:24,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] spmStart polling ended, spm status: SPM >2012-12-17 10:13:24,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e990e37b-fe00-4b7c-a447-7a3953c466d5), log id: 2ff10234 >2012-12-17 10:13:24,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, HSMClearTaskVDSCommand, log id: 2ff10234 >2012-12-17 10:13:24,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [78bc48c3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77c7bc82, log id: 2d3573df >2012-12-17 10:13:24,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:24,462 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:13:24,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 694dc685 >2012-12-17 10:13:24,507 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [78bc48c3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:24,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [200cd224] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:24,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6366f434 >2012-12-17 10:13:24,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 535ee8c5 >2012-12-17 10:13:24,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:24,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, SpmStopVDSCommand, log id: 535ee8c5 >2012-12-17 10:13:24,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, ResetIrsVDSCommand, log id: 6366f434 >2012-12-17 10:13:24,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d5aee25 >2012-12-17 10:13:24,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:24,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d5aee25 >2012-12-17 10:13:25,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:25,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5089 >2012-12-17 10:13:25,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5089, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d22f5d >2012-12-17 10:13:25,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling started: taskId = 7eb9803a-11f6-46fe-b299-3f93f6dea500 >2012-12-17 10:13:26,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling ended: taskId = 7eb9803a-11f6-46fe-b299-3f93f6dea500 task status = finished >2012-12-17 10:13:26,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] spmStart polling ended, spm status: SPM >2012-12-17 10:13:26,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7eb9803a-11f6-46fe-b299-3f93f6dea500), log id: 69cb1d1d >2012-12-17 10:13:26,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMClearTaskVDSCommand, log id: 69cb1d1d >2012-12-17 10:13:26,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@15e4912e, log id: 32d22f5d >2012-12-17 10:13:26,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:26,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:26,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79c0c7e4 >2012-12-17 10:13:26,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79c0c7e4 >2012-12-17 10:13:26,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 694dc685 >2012-12-17 10:13:26,428 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:26,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f83882e >2012-12-17 10:13:26,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:26,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6eb77325 >2012-12-17 10:13:26,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6eb77325 >2012-12-17 10:13:26,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-22) [42fa3e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f83882e >2012-12-17 10:13:26,448 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-22) [42fa3e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:29,681 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:35,120 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1b1c5c6d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:35,361 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [46ee245a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:35,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 121710b5 >2012-12-17 10:13:35,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a73330f >2012-12-17 10:13:35,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:35,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, SpmStopVDSCommand, log id: 7a73330f >2012-12-17 10:13:35,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, ResetIrsVDSCommand, log id: 121710b5 >2012-12-17 10:13:35,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23578c64 >2012-12-17 10:13:35,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [46ee245a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:35,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [46ee245a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23578c64 >2012-12-17 10:13:39,903 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:45,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:45,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5090 >2012-12-17 10:13:45,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5090, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 431717ca >2012-12-17 10:13:45,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling started: taskId = 956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca >2012-12-17 10:13:46,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling ended: taskId = 956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca task status = finished >2012-12-17 10:13:46,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] spmStart polling ended, spm status: SPM >2012-12-17 10:13:46,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=956f1e8a-e74c-4a20-bbec-ff4cb4bc2eca), log id: 5da51f60 >2012-12-17 10:13:46,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, HSMClearTaskVDSCommand, log id: 5da51f60 >2012-12-17 10:13:46,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [205b7df7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3af07a8, log id: 431717ca >2012-12-17 10:13:46,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:47,013 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:13:47,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 108fd21f >2012-12-17 10:13:47,048 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [205b7df7] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:47,224 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [5ee48710] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:47,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2b260f29 >2012-12-17 10:13:47,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7f0d64e1 >2012-12-17 10:13:47,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:47,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, SpmStopVDSCommand, log id: 7f0d64e1 >2012-12-17 10:13:47,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, ResetIrsVDSCommand, log id: 2b260f29 >2012-12-17 10:13:47,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b7849e6 >2012-12-17 10:13:47,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:47,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b7849e6 >2012-12-17 10:13:47,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:13:47,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5091 >2012-12-17 10:13:47,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5091, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39d8bbb6 >2012-12-17 10:13:47,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 7ff7293e-0037-44e9-81c5-75d3fb591e6d >2012-12-17 10:13:48,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 7ff7293e-0037-44e9-81c5-75d3fb591e6d task status = finished >2012-12-17 10:13:48,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM >2012-12-17 10:13:48,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ff7293e-0037-44e9-81c5-75d3fb591e6d), log id: 6fad1af8 >2012-12-17 10:13:48,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 6fad1af8 >2012-12-17 10:13:48,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@274233c1, log id: 39d8bbb6 >2012-12-17 10:13:48,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:13:48,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:48,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31d3e2ba >2012-12-17 10:13:48,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31d3e2ba >2012-12-17 10:13:48,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 108fd21f >2012-12-17 10:13:48,718 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:48,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 17e076a2 >2012-12-17 10:13:48,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:48,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 516903d8 >2012-12-17 10:13:48,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 516903d8 >2012-12-17 10:13:48,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17e076a2 >2012-12-17 10:13:48,738 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:13:50,136 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [4d8dc1c4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:13:57,614 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5654bc9d] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:13:57,909 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [1bc6b95c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:13:57,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 7fcab695 >2012-12-17 10:13:57,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2552b646 >2012-12-17 10:13:57,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:58,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, SpmStopVDSCommand, log id: 2552b646 >2012-12-17 10:13:58,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, ResetIrsVDSCommand, log id: 7fcab695 >2012-12-17 10:13:58,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a474e66 >2012-12-17 10:13:58,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1bc6b95c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:13:58,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [1bc6b95c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a474e66 >2012-12-17 10:14:00,342 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [10ed878d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:14:08,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:08,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5092 >2012-12-17 10:14:08,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5092, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 872c360 >2012-12-17 10:14:08,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling started: taskId = a963af9b-1799-41b8-b1d1-bf414ee80b70 >2012-12-17 10:14:09,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling ended: taskId = a963af9b-1799-41b8-b1d1-bf414ee80b70 task status = finished >2012-12-17 10:14:09,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] spmStart polling ended, spm status: SPM >2012-12-17 10:14:09,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a963af9b-1799-41b8-b1d1-bf414ee80b70), log id: 12dd1aa7 >2012-12-17 10:14:09,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, HSMClearTaskVDSCommand, log id: 12dd1aa7 >2012-12-17 10:14:09,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [56306d9f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7200d39d, log id: 872c360 >2012-12-17 10:14:09,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:09,438 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:14:09,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1cc55c4e >2012-12-17 10:14:09,447 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [56306d9f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:14:09,540 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [2747eada] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:14:09,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 44d418dd >2012-12-17 10:14:09,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 331fb3ae >2012-12-17 10:14:09,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:09,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, SpmStopVDSCommand, log id: 331fb3ae >2012-12-17 10:14:09,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, ResetIrsVDSCommand, log id: 44d418dd >2012-12-17 10:14:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [2747eada] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bca3556 >2012-12-17 10:14:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2747eada] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [2747eada] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bca3556 >2012-12-17 10:14:09,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:09,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5093 >2012-12-17 10:14:09,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5093, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4e04ebd5 >2012-12-17 10:14:09,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling started: taskId = 60012190-4cec-4788-ab27-3d523720d397 >2012-12-17 10:14:10,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended: taskId = 60012190-4cec-4788-ab27-3d523720d397 task status = finished >2012-12-17 10:14:10,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] spmStart polling ended, spm status: SPM >2012-12-17 10:14:10,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=60012190-4cec-4788-ab27-3d523720d397), log id: 3531d15d >2012-12-17 10:14:10,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMClearTaskVDSCommand, log id: 3531d15d >2012-12-17 10:14:10,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59a84bf3, log id: 4e04ebd5 >2012-12-17 10:14:11,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [26a9278b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:11,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:11,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 43f13f49 >2012-12-17 10:14:11,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 43f13f49 >2012-12-17 10:14:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1cc55c4e >2012-12-17 10:14:11,394 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:14:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e68fc41 >2012-12-17 10:14:11,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:11,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 350304dc >2012-12-17 10:14:11,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 350304dc >2012-12-17 10:14:11,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-70) [26a9278b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e68fc41 >2012-12-17 10:14:11,413 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-70) [26a9278b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:14:19,907 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [66369950] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:14:20,012 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [4020b36c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:14:20,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a61f606 >2012-12-17 10:14:20,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d750389 >2012-12-17 10:14:20,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:20,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, SpmStopVDSCommand, log id: 7d750389 >2012-12-17 10:14:20,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, ResetIrsVDSCommand, log id: 2a61f606 >2012-12-17 10:14:20,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58d0e627 >2012-12-17 10:14:20,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [4020b36c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:20,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [4020b36c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58d0e627 >2012-12-17 10:14:20,827 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:14:30,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:30,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5094 >2012-12-17 10:14:30,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5094, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4861a6f6 >2012-12-17 10:14:30,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling started: taskId = 20541ac5-775c-466d-8db7-7b29dcec6be9 >2012-12-17 10:14:31,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended: taskId = 20541ac5-775c-466d-8db7-7b29dcec6be9 task status = finished >2012-12-17 10:14:31,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] spmStart polling ended, spm status: SPM >2012-12-17 10:14:31,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=20541ac5-775c-466d-8db7-7b29dcec6be9), log id: 24bf9497 >2012-12-17 10:14:31,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, HSMClearTaskVDSCommand, log id: 24bf9497 >2012-12-17 10:14:31,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [384b399c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1db4585a, log id: 4861a6f6 >2012-12-17 10:14:31,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:31,872 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:14:31,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a5a2463 >2012-12-17 10:14:31,906 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [384b399c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:14:32,089 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [2fe721f0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:14:32,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 43e1f14b >2012-12-17 10:14:32,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a9d6ffe >2012-12-17 10:14:32,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:32,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, SpmStopVDSCommand, log id: 5a9d6ffe >2012-12-17 10:14:32,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, ResetIrsVDSCommand, log id: 43e1f14b >2012-12-17 10:14:32,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3125a2da >2012-12-17 10:14:32,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fe721f0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:32,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fe721f0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3125a2da >2012-12-17 10:14:32,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:32,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5095 >2012-12-17 10:14:32,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5095, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5de5da1f >2012-12-17 10:14:32,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling started: taskId = 928e028b-4b17-40ff-933b-de1a5181d1b5 >2012-12-17 10:14:33,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling ended: taskId = 928e028b-4b17-40ff-933b-de1a5181d1b5 task status = finished >2012-12-17 10:14:33,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] spmStart polling ended, spm status: SPM >2012-12-17 10:14:33,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=928e028b-4b17-40ff-933b-de1a5181d1b5), log id: 7a310499 >2012-12-17 10:14:33,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMClearTaskVDSCommand, log id: 7a310499 >2012-12-17 10:14:33,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2825177b, log id: 5de5da1f >2012-12-17 10:14:33,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:33,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:33,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1d668a3d >2012-12-17 10:14:33,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d668a3d >2012-12-17 10:14:33,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2a5a2463 >2012-12-17 10:14:33,646 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:14:33,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: a439aba >2012-12-17 10:14:33,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:33,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 112e9bdf >2012-12-17 10:14:33,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 112e9bdf >2012-12-17 10:14:33,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-63) [15d070bf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a439aba >2012-12-17 10:14:33,666 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-63) [15d070bf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:14:42,093 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:14:42,511 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [a97381c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:14:42,733 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [78c0579f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:14:42,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4fd9d3c >2012-12-17 10:14:42,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e87adb0 >2012-12-17 10:14:42,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:43,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, SpmStopVDSCommand, log id: 1e87adb0 >2012-12-17 10:14:43,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, ResetIrsVDSCommand, log id: 4fd9d3c >2012-12-17 10:14:43,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50f6f322 >2012-12-17 10:14:43,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [78c0579f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:43,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [78c0579f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50f6f322 >2012-12-17 10:14:52,352 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:14:53,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:53,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5096 >2012-12-17 10:14:53,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5096, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5787bc95 >2012-12-17 10:14:53,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling started: taskId = 262970bc-a8af-4216-9ac4-3701896876ff >2012-12-17 10:14:54,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling ended: taskId = 262970bc-a8af-4216-9ac4-3701896876ff task status = finished >2012-12-17 10:14:54,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] spmStart polling ended, spm status: SPM >2012-12-17 10:14:54,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=262970bc-a8af-4216-9ac4-3701896876ff), log id: 7eda0cc9 >2012-12-17 10:14:54,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, HSMClearTaskVDSCommand, log id: 7eda0cc9 >2012-12-17 10:14:54,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-30) [2a51251c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d7da12f, log id: 5787bc95 >2012-12-17 10:14:54,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:54,328 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:14:54,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 370468db >2012-12-17 10:14:54,373 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [2a51251c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:14:54,593 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [533115ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:14:54,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4e9b044a >2012-12-17 10:14:54,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2db47c5f >2012-12-17 10:14:54,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:54,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, SpmStopVDSCommand, log id: 2db47c5f >2012-12-17 10:14:54,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, ResetIrsVDSCommand, log id: 4e9b044a >2012-12-17 10:14:54,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [533115ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 467665d1 >2012-12-17 10:14:54,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [533115ff] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:54,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [533115ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 467665d1 >2012-12-17 10:14:54,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:14:54,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5097 >2012-12-17 10:14:54,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5097, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ddcacd6 >2012-12-17 10:14:54,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling started: taskId = 02348d43-bd2a-41d1-bbcb-da5b007cd10b >2012-12-17 10:14:56,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended: taskId = 02348d43-bd2a-41d1-bbcb-da5b007cd10b task status = finished >2012-12-17 10:14:56,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] spmStart polling ended, spm status: SPM >2012-12-17 10:14:56,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=02348d43-bd2a-41d1-bbcb-da5b007cd10b), log id: 25eef2d4 >2012-12-17 10:14:56,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMClearTaskVDSCommand, log id: 25eef2d4 >2012-12-17 10:14:56,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a884fc0, log id: 1ddcacd6 >2012-12-17 10:14:56,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [536ede7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:14:56,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:56,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 78347a2 >2012-12-17 10:14:56,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78347a2 >2012-12-17 10:14:56,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 370468db >2012-12-17 10:14:56,510 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:14:56,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bd3e088 >2012-12-17 10:14:56,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:14:56,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5b04c78e >2012-12-17 10:14:56,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b04c78e >2012-12-17 10:14:56,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-8) [536ede7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bd3e088 >2012-12-17 10:14:56,531 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-8) [536ede7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Checking autorecoverable hosts >2012-12-17 10:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Autorecovering 0 hosts >2012-12-17 10:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Checking autorecoverable hosts done >2012-12-17 10:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4e8c7167] Autorecovering storage domains is disabled, skipping >2012-12-17 10:15:02,554 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [515f648d] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:15:04,974 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [3ced8e0c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:15:05,056 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:15:05,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 46ea23f4 >2012-12-17 10:15:05,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 60c8e2ac >2012-12-17 10:15:05,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:05,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, SpmStopVDSCommand, log id: 60c8e2ac >2012-12-17 10:15:05,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, ResetIrsVDSCommand, log id: 46ea23f4 >2012-12-17 10:15:05,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c7f0b09 >2012-12-17 10:15:05,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:05,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c7f0b09 >2012-12-17 10:15:13,141 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:15:15,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:15:15,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5098 >2012-12-17 10:15:15,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5098, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5283430 >2012-12-17 10:15:15,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling started: taskId = 4a4b71df-6f27-4e1e-88d5-48b1973c1b69 >2012-12-17 10:15:16,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling ended: taskId = 4a4b71df-6f27-4e1e-88d5-48b1973c1b69 task status = finished >2012-12-17 10:15:16,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] spmStart polling ended, spm status: SPM >2012-12-17 10:15:16,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4a4b71df-6f27-4e1e-88d5-48b1973c1b69), log id: 36a2d1a3 >2012-12-17 10:15:16,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, HSMClearTaskVDSCommand, log id: 36a2d1a3 >2012-12-17 10:15:16,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [7c8941c5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b71405a, log id: 5283430 >2012-12-17 10:15:16,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:15:17,073 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:15:17,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13c0d10d >2012-12-17 10:15:17,124 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7c8941c5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:15:17,331 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [6e88e48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:15:17,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1432b1 >2012-12-17 10:15:17,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f1cc073 >2012-12-17 10:15:17,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:17,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, SpmStopVDSCommand, log id: 4f1cc073 >2012-12-17 10:15:17,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, ResetIrsVDSCommand, log id: 1432b1 >2012-12-17 10:15:17,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17677baa >2012-12-17 10:15:17,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [6e88e48] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:17,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [6e88e48] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17677baa >2012-12-17 10:15:17,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:15:17,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5099 >2012-12-17 10:15:17,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5099, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68912f85 >2012-12-17 10:15:17,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling started: taskId = dc7ea60f-0932-45b3-9c1b-554cb22b5560 >2012-12-17 10:15:18,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended: taskId = dc7ea60f-0932-45b3-9c1b-554cb22b5560 task status = finished >2012-12-17 10:15:18,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] spmStart polling ended, spm status: SPM >2012-12-17 10:15:18,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc7ea60f-0932-45b3-9c1b-554cb22b5560), log id: 70730870 >2012-12-17 10:15:18,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMClearTaskVDSCommand, log id: 70730870 >2012-12-17 10:15:18,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@102f2abd, log id: 68912f85 >2012-12-17 10:15:18,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:15:18,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:18,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2c993a7d >2012-12-17 10:15:18,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c993a7d >2012-12-17 10:15:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 13c0d10d >2012-12-17 10:15:18,833 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:15:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7fcefb6a >2012-12-17 10:15:18,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:18,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3444cdf7 >2012-12-17 10:15:18,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3444cdf7 >2012-12-17 10:15:18,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-51) [56fd87e7] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fcefb6a >2012-12-17 10:15:18,853 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) [56fd87e7] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:15:23,340 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:15:27,699 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [39a6c2d3] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:15:27,957 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [5cdf4906] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:15:28,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 45f1ed >2012-12-17 10:15:28,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5e4623d2 >2012-12-17 10:15:28,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:28,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, SpmStopVDSCommand, log id: 5e4623d2 >2012-12-17 10:15:28,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, ResetIrsVDSCommand, log id: 45f1ed >2012-12-17 10:15:28,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3215811c >2012-12-17 10:15:28,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [5cdf4906] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:28,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [5cdf4906] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3215811c >2012-12-17 10:15:33,555 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:15:38,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:15:38,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5100 >2012-12-17 10:15:38,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5100, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a20c5c1 >2012-12-17 10:15:38,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling started: taskId = 6f06bcc6-a5b7-418a-8d78-539da367b99c >2012-12-17 10:15:39,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling ended: taskId = 6f06bcc6-a5b7-418a-8d78-539da367b99c task status = finished >2012-12-17 10:15:39,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] spmStart polling ended, spm status: SPM >2012-12-17 10:15:39,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=6f06bcc6-a5b7-418a-8d78-539da367b99c), log id: 410b5d00 >2012-12-17 10:15:39,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, HSMClearTaskVDSCommand, log id: 410b5d00 >2012-12-17 10:15:39,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5106cc5c, log id: a20c5c1 >2012-12-17 10:15:39,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:15:39,498 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:15:39,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a411948 >2012-12-17 10:15:39,510 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [5b2bc1dd] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:15:39,718 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [32d3383b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:15:39,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 595a5542 >2012-12-17 10:15:39,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6495b2c >2012-12-17 10:15:39,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:39,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, SpmStopVDSCommand, log id: 6495b2c >2012-12-17 10:15:39,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, ResetIrsVDSCommand, log id: 595a5542 >2012-12-17 10:15:39,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 461c963c >2012-12-17 10:15:39,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32d3383b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:39,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32d3383b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 461c963c >2012-12-17 10:15:40,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:15:40,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5101 >2012-12-17 10:15:40,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5101, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29ddcd3c >2012-12-17 10:15:40,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling started: taskId = 8762ecbd-ac43-4da8-9d1f-7adc03ba7589 >2012-12-17 10:15:41,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling ended: taskId = 8762ecbd-ac43-4da8-9d1f-7adc03ba7589 task status = finished >2012-12-17 10:15:41,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] spmStart polling ended, spm status: SPM >2012-12-17 10:15:41,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8762ecbd-ac43-4da8-9d1f-7adc03ba7589), log id: 576a851e >2012-12-17 10:15:41,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMClearTaskVDSCommand, log id: 576a851e >2012-12-17 10:15:41,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1249a168, log id: 29ddcd3c >2012-12-17 10:15:41,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:15:41,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:41,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cb77a3f >2012-12-17 10:15:41,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cb77a3f >2012-12-17 10:15:41,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a411948 >2012-12-17 10:15:41,617 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:15:41,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21bae572 >2012-12-17 10:15:41,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:41,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 617505aa >2012-12-17 10:15:41,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 617505aa >2012-12-17 10:15:41,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-15) [30ddd635] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21bae572 >2012-12-17 10:15:41,637 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-15) [30ddd635] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:15:43,752 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:15:50,107 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [33cde09c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:15:50,237 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [5c1b0caf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:15:50,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 687b4483 >2012-12-17 10:15:50,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e97a3b1 >2012-12-17 10:15:50,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:50,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SpmStopVDSCommand, log id: 1e97a3b1 >2012-12-17 10:15:50,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, ResetIrsVDSCommand, log id: 687b4483 >2012-12-17 10:15:50,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7deaa4e2 >2012-12-17 10:15:50,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:15:50,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7deaa4e2 >2012-12-17 10:15:54,066 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:00,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:00,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5102 >2012-12-17 10:16:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5102, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f5fb8ca >2012-12-17 10:16:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling started: taskId = b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5 >2012-12-17 10:16:01,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling ended: taskId = b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5 task status = finished >2012-12-17 10:16:01,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] spmStart polling ended, spm status: SPM >2012-12-17 10:16:01,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3c648024] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b7eee7cb-c8eb-44b1-ae06-cc8c5c41dea5), log id: 2a72dbe3 >2012-12-17 10:16:01,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, HSMClearTaskVDSCommand, log id: 2a72dbe3 >2012-12-17 10:16:01,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3c648024] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ac229da, log id: 7f5fb8ca >2012-12-17 10:16:01,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:01,840 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:16:01,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5219c606 >2012-12-17 10:16:01,862 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3c648024] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:01,958 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [69c9c5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:02,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2db609e5 >2012-12-17 10:16:02,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: c0501bc >2012-12-17 10:16:02,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:02,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, SpmStopVDSCommand, log id: c0501bc >2012-12-17 10:16:02,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, ResetIrsVDSCommand, log id: 2db609e5 >2012-12-17 10:16:02,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 79ae179c >2012-12-17 10:16:02,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [69c9c5a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:02,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [69c9c5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 79ae179c >2012-12-17 10:16:02,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:02,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5103 >2012-12-17 10:16:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5103, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ca586c4 >2012-12-17 10:16:02,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = 2b8934c8-b5d0-4e48-b529-a2f155fb83db >2012-12-17 10:16:03,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = 2b8934c8-b5d0-4e48-b529-a2f155fb83db task status = finished >2012-12-17 10:16:03,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM >2012-12-17 10:16:03,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b8934c8-b5d0-4e48-b529-a2f155fb83db), log id: 7b33ef9 >2012-12-17 10:16:03,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 7b33ef9 >2012-12-17 10:16:03,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8eb5162, log id: 4ca586c4 >2012-12-17 10:16:03,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:03,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:03,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31195e2f >2012-12-17 10:16:03,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31195e2f >2012-12-17 10:16:03,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5219c606 >2012-12-17 10:16:03,444 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:03,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 787678d7 >2012-12-17 10:16:03,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:03,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 72dfdbda >2012-12-17 10:16:03,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72dfdbda >2012-12-17 10:16:03,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 787678d7 >2012-12-17 10:16:03,465 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) [26d9e89a] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:04,302 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:12,314 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [46423891] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:12,449 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [79d25c3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:12,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2a201b58 >2012-12-17 10:16:12,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: da7943d >2012-12-17 10:16:12,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:13,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SpmStopVDSCommand, log id: da7943d >2012-12-17 10:16:13,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, ResetIrsVDSCommand, log id: 2a201b58 >2012-12-17 10:16:13,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3ba0d7dd >2012-12-17 10:16:13,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:13,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3ba0d7dd >2012-12-17 10:16:14,577 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:23,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:23,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5104 >2012-12-17 10:16:23,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5104, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f6700c2 >2012-12-17 10:16:23,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling started: taskId = d8d4633d-4ea2-4d9d-bd43-950974be2c30 >2012-12-17 10:16:24,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling ended: taskId = d8d4633d-4ea2-4d9d-bd43-950974be2c30 task status = finished >2012-12-17 10:16:24,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] spmStart polling ended, spm status: SPM >2012-12-17 10:16:24,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d8d4633d-4ea2-4d9d-bd43-950974be2c30), log id: 430ab48 >2012-12-17 10:16:24,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, HSMClearTaskVDSCommand, log id: 430ab48 >2012-12-17 10:16:24,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [29a5a9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@173d061e, log id: 3f6700c2 >2012-12-17 10:16:24,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:24,334 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:16:24,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 69efb109 >2012-12-17 10:16:24,358 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [29a5a9b6] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:24,648 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [13905503] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:24,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 16e82304 >2012-12-17 10:16:24,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5a48c4d1 >2012-12-17 10:16:24,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:24,825 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:24,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, SpmStopVDSCommand, log id: 5a48c4d1 >2012-12-17 10:16:24,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, ResetIrsVDSCommand, log id: 16e82304 >2012-12-17 10:16:24,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [13905503] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6384100c >2012-12-17 10:16:24,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [13905503] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:24,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [13905503] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6384100c >2012-12-17 10:16:24,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:25,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5105 >2012-12-17 10:16:25,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5105, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f5eb76e >2012-12-17 10:16:25,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling started: taskId = 8ca9c49b-7e4b-4019-96e7-f2fb114437e9 >2012-12-17 10:16:26,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling ended: taskId = 8ca9c49b-7e4b-4019-96e7-f2fb114437e9 task status = finished >2012-12-17 10:16:26,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] spmStart polling ended, spm status: SPM >2012-12-17 10:16:26,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=8ca9c49b-7e4b-4019-96e7-f2fb114437e9), log id: 2483dfea >2012-12-17 10:16:26,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMClearTaskVDSCommand, log id: 2483dfea >2012-12-17 10:16:26,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4de18e60, log id: 6f5eb76e >2012-12-17 10:16:26,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:26,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:26,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4d90002a >2012-12-17 10:16:26,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d90002a >2012-12-17 10:16:26,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69efb109 >2012-12-17 10:16:26,879 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:26,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28faed2e >2012-12-17 10:16:26,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:26,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 520f8da0 >2012-12-17 10:16:26,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 520f8da0 >2012-12-17 10:16:26,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-19) [3ad075f5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28faed2e >2012-12-17 10:16:26,900 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-19) [3ad075f5] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:26,940 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:16:26,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3545dd1e >2012-12-17 10:16:26,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:26,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1f3e6e23 >2012-12-17 10:16:26,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1f3e6e23 >2012-12-17 10:16:26,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3545dd1e >2012-12-17 10:16:26,960 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:26,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3bb0cbdb >2012-12-17 10:16:26,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:26,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 77eb9fff >2012-12-17 10:16:26,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 77eb9fff >2012-12-17 10:16:26,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3bb0cbdb >2012-12-17 10:16:26,981 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:35,076 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [6038c0b1] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:35,233 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [3cb6b88f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:35,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5ec3305b >2012-12-17 10:16:35,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6a8d0aca >2012-12-17 10:16:35,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:35,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SpmStopVDSCommand, log id: 6a8d0aca >2012-12-17 10:16:35,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, ResetIrsVDSCommand, log id: 5ec3305b >2012-12-17 10:16:35,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ed5db70 >2012-12-17 10:16:35,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:35,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ed5db70 >2012-12-17 10:16:45,314 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:45,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:45,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5106 >2012-12-17 10:16:45,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5106, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 739b3cb6 >2012-12-17 10:16:45,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling started: taskId = b5e1cda6-0ff2-4850-87e1-9465105daad2 >2012-12-17 10:16:46,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling ended: taskId = b5e1cda6-0ff2-4850-87e1-9465105daad2 task status = finished >2012-12-17 10:16:46,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] spmStart polling ended, spm status: SPM >2012-12-17 10:16:46,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b5e1cda6-0ff2-4850-87e1-9465105daad2), log id: 74af36e1 >2012-12-17 10:16:46,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, HSMClearTaskVDSCommand, log id: 74af36e1 >2012-12-17 10:16:46,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [40e7f581] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b3c29a0, log id: 739b3cb6 >2012-12-17 10:16:46,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:46,849 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:16:46,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36812675 >2012-12-17 10:16:46,893 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [40e7f581] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:47,053 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [127fd955] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:47,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 66bb91f1 >2012-12-17 10:16:47,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5af5c697 >2012-12-17 10:16:47,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:47,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, SpmStopVDSCommand, log id: 5af5c697 >2012-12-17 10:16:47,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, ResetIrsVDSCommand, log id: 66bb91f1 >2012-12-17 10:16:47,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [127fd955] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 744a1f62 >2012-12-17 10:16:47,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [127fd955] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:47,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [127fd955] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 744a1f62 >2012-12-17 10:16:47,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:16:47,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5107 >2012-12-17 10:16:47,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5107, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36a656ae >2012-12-17 10:16:47,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling started: taskId = c606391d-35de-4551-9519-993dd589a644 >2012-12-17 10:16:48,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended: taskId = c606391d-35de-4551-9519-993dd589a644 task status = finished >2012-12-17 10:16:48,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] spmStart polling ended, spm status: SPM >2012-12-17 10:16:48,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=c606391d-35de-4551-9519-993dd589a644), log id: 655480d >2012-12-17 10:16:48,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMClearTaskVDSCommand, log id: 655480d >2012-12-17 10:16:48,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48ed50ba, log id: 36a656ae >2012-12-17 10:16:48,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [2e31d140] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:16:48,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:48,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 71a206a7 >2012-12-17 10:16:48,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 71a206a7 >2012-12-17 10:16:48,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36812675 >2012-12-17 10:16:48,494 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:48,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d968942 >2012-12-17 10:16:48,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:48,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4b85a9e9 >2012-12-17 10:16:48,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b85a9e9 >2012-12-17 10:16:48,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-28) [2e31d140] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d968942 >2012-12-17 10:16:48,514 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-28) [2e31d140] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:16:55,599 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [56fd87e7] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:16:57,363 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [78864454] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:16:57,479 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [454eb5a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:16:57,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 569acbba >2012-12-17 10:16:57,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2bbe0d29 >2012-12-17 10:16:57,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:57,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, SpmStopVDSCommand, log id: 2bbe0d29 >2012-12-17 10:16:57,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, ResetIrsVDSCommand, log id: 569acbba >2012-12-17 10:16:57,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6888cc13 >2012-12-17 10:16:57,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [454eb5a9] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:16:57,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [454eb5a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6888cc13 >2012-12-17 10:17:05,813 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:17:07,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:07,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5108 >2012-12-17 10:17:07,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5108, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 395a16a0 >2012-12-17 10:17:07,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling started: taskId = 589c6560-0846-4894-bdb7-7cd929d3c2c1 >2012-12-17 10:17:08,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling ended: taskId = 589c6560-0846-4894-bdb7-7cd929d3c2c1 task status = finished >2012-12-17 10:17:08,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] spmStart polling ended, spm status: SPM >2012-12-17 10:17:08,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=589c6560-0846-4894-bdb7-7cd929d3c2c1), log id: 173d7d1c >2012-12-17 10:17:08,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, HSMClearTaskVDSCommand, log id: 173d7d1c >2012-12-17 10:17:08,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [a355c3f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45e0c52f, log id: 395a16a0 >2012-12-17 10:17:09,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:09,045 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:17:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d6176a5 >2012-12-17 10:17:09,074 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [a355c3f] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:17:09,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [57fa1ec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:17:09,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4488b9a8 >2012-12-17 10:17:09,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 41dbaf69 >2012-12-17 10:17:09,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:09,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, SpmStopVDSCommand, log id: 41dbaf69 >2012-12-17 10:17:09,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, ResetIrsVDSCommand, log id: 4488b9a8 >2012-12-17 10:17:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24b3e4f1 >2012-12-17 10:17:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [57fa1ec] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [57fa1ec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24b3e4f1 >2012-12-17 10:17:09,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:09,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5109 >2012-12-17 10:17:09,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5109, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3dfcc9ee >2012-12-17 10:17:09,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling started: taskId = 7ba80757-2150-4dbe-93d3-4028a1c17260 >2012-12-17 10:17:10,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended: taskId = 7ba80757-2150-4dbe-93d3-4028a1c17260 task status = finished >2012-12-17 10:17:10,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] spmStart polling ended, spm status: SPM >2012-12-17 10:17:10,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7ba80757-2150-4dbe-93d3-4028a1c17260), log id: 3a286dd >2012-12-17 10:17:10,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMClearTaskVDSCommand, log id: 3a286dd >2012-12-17 10:17:10,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23441908, log id: 3dfcc9ee >2012-12-17 10:17:10,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:10,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:10,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 60f10208 >2012-12-17 10:17:10,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 60f10208 >2012-12-17 10:17:10,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: d6176a5 >2012-12-17 10:17:10,699 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:10,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6ebc8cf0 >2012-12-17 10:17:10,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:10,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3931a871 >2012-12-17 10:17:10,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3931a871 >2012-12-17 10:17:10,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [5c496b37] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6ebc8cf0 >2012-12-17 10:17:10,720 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [5c496b37] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:16,892 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:17:19,600 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [2747eada] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:17:19,708 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [780047c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:17:19,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 4097a227 >2012-12-17 10:17:19,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 359a8573 >2012-12-17 10:17:19,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:19,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SpmStopVDSCommand, log id: 359a8573 >2012-12-17 10:17:20,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, ResetIrsVDSCommand, log id: 4097a227 >2012-12-17 10:17:20,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e797fda >2012-12-17 10:17:20,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [780047c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:20,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e797fda >2012-12-17 10:17:27,126 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:17:30,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:30,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5110 >2012-12-17 10:17:30,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5110, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22a9f199 >2012-12-17 10:17:30,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling started: taskId = e878e899-6a23-489b-bd97-546667c6aba6 >2012-12-17 10:17:31,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling ended: taskId = e878e899-6a23-489b-bd97-546667c6aba6 task status = finished >2012-12-17 10:17:31,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] spmStart polling ended, spm status: SPM >2012-12-17 10:17:31,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e878e899-6a23-489b-bd97-546667c6aba6), log id: 4b21677f >2012-12-17 10:17:31,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, HSMClearTaskVDSCommand, log id: 4b21677f >2012-12-17 10:17:31,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [fc992c9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e11e961, log id: 22a9f199 >2012-12-17 10:17:31,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:31,880 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:17:31,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49f5f158 >2012-12-17 10:17:31,895 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [fc992c9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:17:32,072 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [938c7ef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:17:32,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 75a27495 >2012-12-17 10:17:32,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5f71cc96 >2012-12-17 10:17:32,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:32,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, SpmStopVDSCommand, log id: 5f71cc96 >2012-12-17 10:17:32,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, ResetIrsVDSCommand, log id: 75a27495 >2012-12-17 10:17:32,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e4bce3 >2012-12-17 10:17:32,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [938c7ef] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:32,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [938c7ef] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e4bce3 >2012-12-17 10:17:32,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:32,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5111 >2012-12-17 10:17:32,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5111, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fdf7b37 >2012-12-17 10:17:32,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling started: taskId = f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855 >2012-12-17 10:17:33,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling ended: taskId = f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855 task status = finished >2012-12-17 10:17:33,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] spmStart polling ended, spm status: SPM >2012-12-17 10:17:33,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f1f1d1e0-bfaf-4f83-85d7-8dc8a8447855), log id: 7e48ee5d >2012-12-17 10:17:33,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMClearTaskVDSCommand, log id: 7e48ee5d >2012-12-17 10:17:33,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@66195c70, log id: 1fdf7b37 >2012-12-17 10:17:33,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:33,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:33,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63f97ca1 >2012-12-17 10:17:33,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63f97ca1 >2012-12-17 10:17:33,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49f5f158 >2012-12-17 10:17:33,622 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:33,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fd7504a >2012-12-17 10:17:33,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:33,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 4cced4a1 >2012-12-17 10:17:33,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cced4a1 >2012-12-17 10:17:33,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-35) [141038b9] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6fd7504a >2012-12-17 10:17:33,642 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-35) [141038b9] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:37,439 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [7dff3777] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:17:42,535 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fe721f0] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:17:42,699 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [17997e65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:17:42,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 17a1bcad >2012-12-17 10:17:42,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: e415e67 >2012-12-17 10:17:42,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:42,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, SpmStopVDSCommand, log id: e415e67 >2012-12-17 10:17:42,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, ResetIrsVDSCommand, log id: 17a1bcad >2012-12-17 10:17:42,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71e1794e >2012-12-17 10:17:42,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:42,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71e1794e >2012-12-17 10:17:47,659 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:17:53,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:53,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5112 >2012-12-17 10:17:53,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5112, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73cf0ae0 >2012-12-17 10:17:53,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling started: taskId = 1df34274-a1cd-41d5-a943-acb5d65200aa >2012-12-17 10:17:54,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended: taskId = 1df34274-a1cd-41d5-a943-acb5d65200aa task status = finished >2012-12-17 10:17:54,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] spmStart polling ended, spm status: SPM >2012-12-17 10:17:54,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1df34274-a1cd-41d5-a943-acb5d65200aa), log id: 7090aaf >2012-12-17 10:17:54,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, HSMClearTaskVDSCommand, log id: 7090aaf >2012-12-17 10:17:54,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [75689c2c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@629c6ac4, log id: 73cf0ae0 >2012-12-17 10:17:54,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:54,193 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:17:54,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4fc34169 >2012-12-17 10:17:54,251 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [75689c2c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:17:54,437 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-54) [44e35e2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:17:54,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: fe9b236 >2012-12-17 10:17:54,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 9152f53 >2012-12-17 10:17:54,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:54,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, SpmStopVDSCommand, log id: 9152f53 >2012-12-17 10:17:54,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, ResetIrsVDSCommand, log id: fe9b236 >2012-12-17 10:17:54,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a656cc9 >2012-12-17 10:17:54,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [44e35e2b] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:54,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-54) [44e35e2b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a656cc9 >2012-12-17 10:17:54,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:17:54,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5113 >2012-12-17 10:17:54,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5113, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e3373b9 >2012-12-17 10:17:54,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling started: taskId = bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6 >2012-12-17 10:17:55,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling ended: taskId = bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6 task status = finished >2012-12-17 10:17:55,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] spmStart polling ended, spm status: SPM >2012-12-17 10:17:55,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bd72d14a-bbc3-42f0-9ca0-9f7a6e10e4c6), log id: 5ce31ea5 >2012-12-17 10:17:55,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMClearTaskVDSCommand, log id: 5ce31ea5 >2012-12-17 10:17:55,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42f11247, log id: e3373b9 >2012-12-17 10:17:56,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [55bb30e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:17:56,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:56,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 63a12f6e >2012-12-17 10:17:56,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 63a12f6e >2012-12-17 10:17:56,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fc34169 >2012-12-17 10:17:56,235 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:56,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6b3111db >2012-12-17 10:17:56,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:17:56,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7e24ea83 >2012-12-17 10:17:56,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e24ea83 >2012-12-17 10:17:56,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-69) [55bb30e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b3111db >2012-12-17 10:17:56,266 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) [55bb30e] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:17:57,877 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [3da48235] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:18:04,856 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [302432ce] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:18:04,945 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [e1a392] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:18:04,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 532823bc >2012-12-17 10:18:04,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1e673eb5 >2012-12-17 10:18:05,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:05,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, SpmStopVDSCommand, log id: 1e673eb5 >2012-12-17 10:18:05,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, ResetIrsVDSCommand, log id: 532823bc >2012-12-17 10:18:05,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 573408d6 >2012-12-17 10:18:05,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:05,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 573408d6 >2012-12-17 10:18:08,144 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [780047c3] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:18:15,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:18:15,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5114 >2012-12-17 10:18:15,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5114, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55625bea >2012-12-17 10:18:15,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling started: taskId = 7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81 >2012-12-17 10:18:16,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended: taskId = 7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81 task status = finished >2012-12-17 10:18:16,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] spmStart polling ended, spm status: SPM >2012-12-17 10:18:16,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7e24c51d-1a32-4a8e-bdc4-9c32d6e04b81), log id: 793c7cd0 >2012-12-17 10:18:16,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, HSMClearTaskVDSCommand, log id: 793c7cd0 >2012-12-17 10:18:16,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [346a0820] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@722679de, log id: 55625bea >2012-12-17 10:18:16,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:18:16,907 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:18:16,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 73bbcf2c >2012-12-17 10:18:16,930 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [346a0820] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:18:17,176 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [6431f17a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:18:17,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 666a6f3d >2012-12-17 10:18:17,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3bb9e08 >2012-12-17 10:18:17,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:17,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, SpmStopVDSCommand, log id: 3bb9e08 >2012-12-17 10:18:17,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, ResetIrsVDSCommand, log id: 666a6f3d >2012-12-17 10:18:17,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48c9e822 >2012-12-17 10:18:17,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [6431f17a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:17,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [6431f17a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48c9e822 >2012-12-17 10:18:17,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:18:17,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5115 >2012-12-17 10:18:17,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5115, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cfdf2ec >2012-12-17 10:18:17,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling started: taskId = 73c8c1ff-a6a3-4be3-95ca-ead5f566d244 >2012-12-17 10:18:18,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling ended: taskId = 73c8c1ff-a6a3-4be3-95ca-ead5f566d244 task status = finished >2012-12-17 10:18:18,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] spmStart polling ended, spm status: SPM >2012-12-17 10:18:18,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=73c8c1ff-a6a3-4be3-95ca-ead5f566d244), log id: 5dc40eeb >2012-12-17 10:18:18,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMClearTaskVDSCommand, log id: 5dc40eeb >2012-12-17 10:18:18,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72af9664, log id: 6cfdf2ec >2012-12-17 10:18:18,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:18:18,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:18,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31731b61 >2012-12-17 10:18:18,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31731b61 >2012-12-17 10:18:18,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 73bbcf2c >2012-12-17 10:18:18,646 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:18:18,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1b27e6f4 >2012-12-17 10:18:18,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:18,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 5182656e >2012-12-17 10:18:18,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5182656e >2012-12-17 10:18:18,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-97) [68ede0fd] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1b27e6f4 >2012-12-17 10:18:18,667 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-97) [68ede0fd] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:18:27,515 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2420ee59] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:18:27,630 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [25e2aebf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:18:27,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1d98384 >2012-12-17 10:18:27,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 316cd540 >2012-12-17 10:18:27,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:27,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, SpmStopVDSCommand, log id: 316cd540 >2012-12-17 10:18:27,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, ResetIrsVDSCommand, log id: 1d98384 >2012-12-17 10:18:27,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bd470ec >2012-12-17 10:18:27,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [25e2aebf] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:27,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [25e2aebf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bd470ec >2012-12-17 10:18:28,533 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:18:37,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:18:38,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5116 >2012-12-17 10:18:38,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5116, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bcea1c9 >2012-12-17 10:18:38,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling started: taskId = 7cb350bc-a383-4d21-a5a1-117b0ed6c656 >2012-12-17 10:18:39,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended: taskId = 7cb350bc-a383-4d21-a5a1-117b0ed6c656 task status = finished >2012-12-17 10:18:39,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] spmStart polling ended, spm status: SPM >2012-12-17 10:18:39,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=7cb350bc-a383-4d21-a5a1-117b0ed6c656), log id: 3c4ae40f >2012-12-17 10:18:39,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, HSMClearTaskVDSCommand, log id: 3c4ae40f >2012-12-17 10:18:39,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5c1c47fa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@58c2b3b5, log id: 1bcea1c9 >2012-12-17 10:18:39,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:18:39,174 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:18:39,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7f0dca0a >2012-12-17 10:18:39,185 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5c1c47fa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:18:39,377 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [3c8f0aba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:18:39,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: eaa26d6 >2012-12-17 10:18:39,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2e3a94b9 >2012-12-17 10:18:39,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:39,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, SpmStopVDSCommand, log id: 2e3a94b9 >2012-12-17 10:18:39,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, ResetIrsVDSCommand, log id: eaa26d6 >2012-12-17 10:18:39,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28c3a303 >2012-12-17 10:18:39,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [3c8f0aba] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:39,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [3c8f0aba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28c3a303 >2012-12-17 10:18:39,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:18:39,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5117 >2012-12-17 10:18:39,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5117, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31580d36 >2012-12-17 10:18:39,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling started: taskId = 1b16cf0e-80d0-4f4a-bfa2-8076804ce07d >2012-12-17 10:18:40,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended: taskId = 1b16cf0e-80d0-4f4a-bfa2-8076804ce07d task status = finished >2012-12-17 10:18:40,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] spmStart polling ended, spm status: SPM >2012-12-17 10:18:40,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1b16cf0e-80d0-4f4a-bfa2-8076804ce07d), log id: 395a8525 >2012-12-17 10:18:40,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMClearTaskVDSCommand, log id: 395a8525 >2012-12-17 10:18:40,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65db5d8d, log id: 31580d36 >2012-12-17 10:18:41,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:18:41,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:41,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1ba23bd9 >2012-12-17 10:18:41,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ba23bd9 >2012-12-17 10:18:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f0dca0a >2012-12-17 10:18:41,793 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:18:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 380731bb >2012-12-17 10:18:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:41,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 109eab13 >2012-12-17 10:18:41,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 109eab13 >2012-12-17 10:18:41,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-3) [7646beaa] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 380731bb >2012-12-17 10:18:41,813 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-3) [7646beaa] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:18:48,994 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [18826afe] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:18:49,938 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [3ad075f5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:18:50,080 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [4fc51f8f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:18:50,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 140d8ac6 >2012-12-17 10:18:50,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1fc90ff4 >2012-12-17 10:18:50,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:50,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, SpmStopVDSCommand, log id: 1fc90ff4 >2012-12-17 10:18:50,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, ResetIrsVDSCommand, log id: 140d8ac6 >2012-12-17 10:18:50,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3940f4ae >2012-12-17 10:18:50,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [4fc51f8f] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:18:50,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [4fc51f8f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3940f4ae >2012-12-17 10:18:59,225 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [4e8c7167] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:00,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:00,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5118 >2012-12-17 10:19:00,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5118, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f894981 >2012-12-17 10:19:00,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling started: taskId = e804ced3-95f5-47f9-81fc-31b0cb50506c >2012-12-17 10:19:01,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling ended: taskId = e804ced3-95f5-47f9-81fc-31b0cb50506c task status = finished >2012-12-17 10:19:01,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] spmStart polling ended, spm status: SPM >2012-12-17 10:19:01,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6001c862] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=e804ced3-95f5-47f9-81fc-31b0cb50506c), log id: 12e718a3 >2012-12-17 10:19:01,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, HSMClearTaskVDSCommand, log id: 12e718a3 >2012-12-17 10:19:01,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [6001c862] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2027f07, log id: 7f894981 >2012-12-17 10:19:01,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:01,614 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:19:01,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9d42aec >2012-12-17 10:19:01,639 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6001c862] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:01,741 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [681720a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:01,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25de93d >2012-12-17 10:19:01,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 75eed51f >2012-12-17 10:19:01,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, SpmStopVDSCommand, log id: 75eed51f >2012-12-17 10:19:02,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, ResetIrsVDSCommand, log id: 25de93d >2012-12-17 10:19:02,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [681720a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b788166 >2012-12-17 10:19:02,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [681720a4] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:02,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [681720a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b788166 >2012-12-17 10:19:02,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:02,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5119 >2012-12-17 10:19:02,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5119, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 720928b6 >2012-12-17 10:19:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling started: taskId = 3341a462-6fb3-4ab2-ae1f-fc266c74d36e >2012-12-17 10:19:03,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling ended: taskId = 3341a462-6fb3-4ab2-ae1f-fc266c74d36e task status = finished >2012-12-17 10:19:03,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] spmStart polling ended, spm status: SPM >2012-12-17 10:19:03,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=3341a462-6fb3-4ab2-ae1f-fc266c74d36e), log id: 25553b5d >2012-12-17 10:19:03,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMClearTaskVDSCommand, log id: 25553b5d >2012-12-17 10:19:03,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7eac69d0, log id: 720928b6 >2012-12-17 10:19:03,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5c1b0caf] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:03,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:03,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 59863e4d >2012-12-17 10:19:03,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 59863e4d >2012-12-17 10:19:03,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9d42aec >2012-12-17 10:19:03,315 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:03,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21a2b5f0 >2012-12-17 10:19:03,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:03,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6109e86a >2012-12-17 10:19:03,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6109e86a >2012-12-17 10:19:03,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-49) [5c1b0caf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 21a2b5f0 >2012-12-17 10:19:03,335 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-49) [5c1b0caf] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:09,510 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [141038b9] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:12,174 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [69c9c5a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:12,306 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [36ca77a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:12,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3552fc3e >2012-12-17 10:19:12,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 10c2dea6 >2012-12-17 10:19:12,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:12,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, SpmStopVDSCommand, log id: 10c2dea6 >2012-12-17 10:19:12,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, ResetIrsVDSCommand, log id: 3552fc3e >2012-12-17 10:19:12,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b20adec >2012-12-17 10:19:12,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [36ca77a1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:12,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [36ca77a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b20adec >2012-12-17 10:19:19,951 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [15d070bf] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:22,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:22,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5120 >2012-12-17 10:19:22,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5120, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79199a47 >2012-12-17 10:19:22,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling started: taskId = 49ad612e-458d-4192-aeab-bdad0c7dfc72 >2012-12-17 10:19:23,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling ended: taskId = 49ad612e-458d-4192-aeab-bdad0c7dfc72 task status = finished >2012-12-17 10:19:23,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] spmStart polling ended, spm status: SPM >2012-12-17 10:19:23,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=49ad612e-458d-4192-aeab-bdad0c7dfc72), log id: 4091e65d >2012-12-17 10:19:23,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, HSMClearTaskVDSCommand, log id: 4091e65d >2012-12-17 10:19:23,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [1db2b76] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2aab04c9, log id: 79199a47 >2012-12-17 10:19:23,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:23,983 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:19:23,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c5709e6 >2012-12-17 10:19:24,035 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [1db2b76] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:24,215 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [50119c91] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:24,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 5bdd0bfe >2012-12-17 10:19:24,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 24f8f070 >2012-12-17 10:19:24,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:24,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, SpmStopVDSCommand, log id: 24f8f070 >2012-12-17 10:19:24,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, ResetIrsVDSCommand, log id: 5bdd0bfe >2012-12-17 10:19:24,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50119c91] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a0afb40 >2012-12-17 10:19:24,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [50119c91] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:24,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [50119c91] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a0afb40 >2012-12-17 10:19:24,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:24,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5121 >2012-12-17 10:19:24,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5121, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4eea7534 >2012-12-17 10:19:24,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling started: taskId = 4ad99371-bfc7-4f9a-9433-4505cf7c3ec7 >2012-12-17 10:19:25,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended: taskId = 4ad99371-bfc7-4f9a-9433-4505cf7c3ec7 task status = finished >2012-12-17 10:19:25,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] spmStart polling ended, spm status: SPM >2012-12-17 10:19:25,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=4ad99371-bfc7-4f9a-9433-4505cf7c3ec7), log id: 5665b3b7 >2012-12-17 10:19:25,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMClearTaskVDSCommand, log id: 5665b3b7 >2012-12-17 10:19:25,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2ef9dfb1, log id: 4eea7534 >2012-12-17 10:19:25,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:25,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:25,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 8d99e05 >2012-12-17 10:19:25,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 8d99e05 >2012-12-17 10:19:25,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2c5709e6 >2012-12-17 10:19:25,741 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:25,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7155ecd >2012-12-17 10:19:25,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:25,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 61c7b000 >2012-12-17 10:19:25,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 61c7b000 >2012-12-17 10:19:25,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7155ecd >2012-12-17 10:19:25,762 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-55) [4007c40f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:30,194 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [569d76f1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:34,654 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [5c496b37] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:34,866 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [60aea922] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:34,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6fb341b0 >2012-12-17 10:19:34,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4c906331 >2012-12-17 10:19:34,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:35,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SpmStopVDSCommand, log id: 4c906331 >2012-12-17 10:19:35,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, ResetIrsVDSCommand, log id: 6fb341b0 >2012-12-17 10:19:35,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65a9a53e >2012-12-17 10:19:35,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:35,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65a9a53e >2012-12-17 10:19:40,413 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [533115ff] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:45,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:45,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5122 >2012-12-17 10:19:45,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5122, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44709dd2 >2012-12-17 10:19:45,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling started: taskId = d0dc1c79-b50a-42a3-a66a-ebcd90b5f359 >2012-12-17 10:19:46,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling ended: taskId = d0dc1c79-b50a-42a3-a66a-ebcd90b5f359 task status = finished >2012-12-17 10:19:46,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] spmStart polling ended, spm status: SPM >2012-12-17 10:19:46,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [200cd224] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d0dc1c79-b50a-42a3-a66a-ebcd90b5f359), log id: 115ec9b7 >2012-12-17 10:19:46,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, HSMClearTaskVDSCommand, log id: 115ec9b7 >2012-12-17 10:19:46,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [200cd224] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46ee53f1, log id: 44709dd2 >2012-12-17 10:19:46,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:46,710 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:19:46,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 610e35df >2012-12-17 10:19:46,728 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [200cd224] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:46,876 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [8102db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:46,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 53799e9f >2012-12-17 10:19:46,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7618654b >2012-12-17 10:19:46,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:47,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, SpmStopVDSCommand, log id: 7618654b >2012-12-17 10:19:47,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, ResetIrsVDSCommand, log id: 53799e9f >2012-12-17 10:19:47,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [8102db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4abf0385 >2012-12-17 10:19:47,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [8102db] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:47,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [8102db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4abf0385 >2012-12-17 10:19:47,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:19:47,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5123 >2012-12-17 10:19:47,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5123, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3609d864 >2012-12-17 10:19:47,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling started: taskId = d7728829-bb32-4334-befa-ab2fcd3c50d4 >2012-12-17 10:19:48,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling ended: taskId = d7728829-bb32-4334-befa-ab2fcd3c50d4 task status = finished >2012-12-17 10:19:48,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] spmStart polling ended, spm status: SPM >2012-12-17 10:19:48,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d7728829-bb32-4334-befa-ab2fcd3c50d4), log id: 53e85008 >2012-12-17 10:19:48,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMClearTaskVDSCommand, log id: 53e85008 >2012-12-17 10:19:48,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38de6841, log id: 3609d864 >2012-12-17 10:19:48,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [3cb6b88f] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:19:48,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:48,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 65a4eea4 >2012-12-17 10:19:48,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 65a4eea4 >2012-12-17 10:19:48,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 610e35df >2012-12-17 10:19:48,353 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:48,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58cc8cc5 >2012-12-17 10:19:48,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:48,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7bbd3c10 >2012-12-17 10:19:48,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bbd3c10 >2012-12-17 10:19:48,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-18) [3cb6b88f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 58cc8cc5 >2012-12-17 10:19:48,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) [3cb6b88f] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:19:50,586 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [47513fa0] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:19:57,253 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [17eab1eb] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:19:57,386 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [75377a97] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:19:57,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1dbc60e8 >2012-12-17 10:19:57,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 51a4121a >2012-12-17 10:19:57,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:57,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, SpmStopVDSCommand, log id: 51a4121a >2012-12-17 10:19:57,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, ResetIrsVDSCommand, log id: 1dbc60e8 >2012-12-17 10:19:57,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [75377a97] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24265102 >2012-12-17 10:19:57,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [75377a97] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:19:57,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [75377a97] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24265102 >2012-12-17 10:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Checking autorecoverable hosts >2012-12-17 10:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Autorecovering 0 hosts >2012-12-17 10:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Checking autorecoverable hosts done >2012-12-17 10:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [7dff3777] Autorecovering storage domains is disabled, skipping >2012-12-17 10:20:00,795 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:20:07,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:07,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5124 >2012-12-17 10:20:07,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5124, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5985ce4e >2012-12-17 10:20:07,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling started: taskId = 38e12a43-ffd9-4120-a2b9-848c60396943 >2012-12-17 10:20:08,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended: taskId = 38e12a43-ffd9-4120-a2b9-848c60396943 task status = finished >2012-12-17 10:20:08,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] spmStart polling ended, spm status: SPM >2012-12-17 10:20:08,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=38e12a43-ffd9-4120-a2b9-848c60396943), log id: a1c0486 >2012-12-17 10:20:08,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, HSMClearTaskVDSCommand, log id: a1c0486 >2012-12-17 10:20:08,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [10600189] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@356cfe99, log id: 5985ce4e >2012-12-17 10:20:08,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:08,908 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:20:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6932924c >2012-12-17 10:20:08,949 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [10600189] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:20:09,135 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [2c0da98a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:20:09,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 472c10f8 >2012-12-17 10:20:09,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3645e52d >2012-12-17 10:20:09,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:09,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, SpmStopVDSCommand, log id: 3645e52d >2012-12-17 10:20:09,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, ResetIrsVDSCommand, log id: 472c10f8 >2012-12-17 10:20:09,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34eea7e4 >2012-12-17 10:20:09,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2c0da98a] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:09,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2c0da98a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34eea7e4 >2012-12-17 10:20:09,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:09,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5125 >2012-12-17 10:20:09,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5125, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f95c1e >2012-12-17 10:20:09,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling started: taskId = f9e9ee3a-af98-4826-ac5a-5949472d36e9 >2012-12-17 10:20:10,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling ended: taskId = f9e9ee3a-af98-4826-ac5a-5949472d36e9 task status = finished >2012-12-17 10:20:10,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] spmStart polling ended, spm status: SPM >2012-12-17 10:20:10,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f9e9ee3a-af98-4826-ac5a-5949472d36e9), log id: 770cbb73 >2012-12-17 10:20:10,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMClearTaskVDSCommand, log id: 770cbb73 >2012-12-17 10:20:10,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e6b814, log id: 6f95c1e >2012-12-17 10:20:10,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:10,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:10,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: af38e67 >2012-12-17 10:20:10,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: af38e67 >2012-12-17 10:20:10,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6932924c >2012-12-17 10:20:10,639 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:20:10,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79172b00 >2012-12-17 10:20:10,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:10,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 713b2576 >2012-12-17 10:20:10,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 713b2576 >2012-12-17 10:20:10,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-2) [79d25c3b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 79172b00 >2012-12-17 10:20:10,659 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-2) [79d25c3b] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:20:11,427 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:20:19,503 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [1bc6b95c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:20:19,653 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [65d7a0e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:20:19,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 218a5785 >2012-12-17 10:20:19,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2311b0dd >2012-12-17 10:20:19,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:19,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SpmStopVDSCommand, log id: 2311b0dd >2012-12-17 10:20:19,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, ResetIrsVDSCommand, log id: 218a5785 >2012-12-17 10:20:19,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12b71069 >2012-12-17 10:20:19,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:19,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12b71069 >2012-12-17 10:20:21,878 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [13905503] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:20:30,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:30,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5126 >2012-12-17 10:20:30,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5126, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1a7a8e7a >2012-12-17 10:20:30,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling started: taskId = a3f6f0da-7c97-4ce3-93a3-867b1921fdfd >2012-12-17 10:20:31,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended: taskId = a3f6f0da-7c97-4ce3-93a3-867b1921fdfd task status = finished >2012-12-17 10:20:31,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] spmStart polling ended, spm status: SPM >2012-12-17 10:20:31,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a3f6f0da-7c97-4ce3-93a3-867b1921fdfd), log id: 368c5bb2 >2012-12-17 10:20:31,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, HSMClearTaskVDSCommand, log id: 368c5bb2 >2012-12-17 10:20:31,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) [5fb78147] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38d81f18, log id: 1a7a8e7a >2012-12-17 10:20:31,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:31,997 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:20:31,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28326016 >2012-12-17 10:20:32,018 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [5fb78147] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:20:32,269 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [3f24993] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:20:32,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 3e71a3f8 >2012-12-17 10:20:32,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 13e57dad >2012-12-17 10:20:32,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:32,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, SpmStopVDSCommand, log id: 13e57dad >2012-12-17 10:20:32,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, ResetIrsVDSCommand, log id: 3e71a3f8 >2012-12-17 10:20:32,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63e3e292 >2012-12-17 10:20:32,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [3f24993] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:32,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63e3e292 >2012-12-17 10:20:32,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:32,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5127 >2012-12-17 10:20:32,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5127, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4586519d >2012-12-17 10:20:32,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling started: taskId = 855dd530-fce8-46d8-ac4b-0a9c95002d56 >2012-12-17 10:20:33,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling ended: taskId = 855dd530-fce8-46d8-ac4b-0a9c95002d56 task status = finished >2012-12-17 10:20:33,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] spmStart polling ended, spm status: SPM >2012-12-17 10:20:33,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=855dd530-fce8-46d8-ac4b-0a9c95002d56), log id: 423987cf >2012-12-17 10:20:33,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMClearTaskVDSCommand, log id: 423987cf >2012-12-17 10:20:33,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f034900, log id: 4586519d >2012-12-17 10:20:33,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3d5285e1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:33,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:33,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 24432d9e >2012-12-17 10:20:33,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 24432d9e >2012-12-17 10:20:33,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28326016 >2012-12-17 10:20:33,727 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:20:33,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3667f2ab >2012-12-17 10:20:33,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:33,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75c6da8a >2012-12-17 10:20:33,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75c6da8a >2012-12-17 10:20:33,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-13) [3d5285e1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3667f2ab >2012-12-17 10:20:33,747 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-13) [3d5285e1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:20:42,373 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:20:42,656 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [2353b939] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:20:42,834 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [77748919] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:20:42,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2e770c49 >2012-12-17 10:20:42,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7d5a4ab7 >2012-12-17 10:20:42,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:43,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, SpmStopVDSCommand, log id: 7d5a4ab7 >2012-12-17 10:20:43,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, ResetIrsVDSCommand, log id: 2e770c49 >2012-12-17 10:20:43,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [77748919] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e3a2344 >2012-12-17 10:20:43,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [77748919] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:43,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [77748919] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e3a2344 >2012-12-17 10:20:52,658 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [742a6686] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:20:53,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:53,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5128 >2012-12-17 10:20:53,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5128, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b66d038 >2012-12-17 10:20:53,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling started: taskId = eddc436d-94cd-4bdf-ada2-1fd4040a95ce >2012-12-17 10:20:54,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling ended: taskId = eddc436d-94cd-4bdf-ada2-1fd4040a95ce task status = finished >2012-12-17 10:20:54,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] spmStart polling ended, spm status: SPM >2012-12-17 10:20:54,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [17997e65] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=eddc436d-94cd-4bdf-ada2-1fd4040a95ce), log id: 54af3d68 >2012-12-17 10:20:54,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, HSMClearTaskVDSCommand, log id: 54af3d68 >2012-12-17 10:20:54,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [17997e65] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ce8a247, log id: b66d038 >2012-12-17 10:20:54,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:54,320 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:20:54,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a768402 >2012-12-17 10:20:54,349 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [17997e65] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:20:54,503 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [4201b256] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:20:54,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2aa8f083 >2012-12-17 10:20:54,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 3134b1ca >2012-12-17 10:20:54,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:54,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, SpmStopVDSCommand, log id: 3134b1ca >2012-12-17 10:20:54,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, ResetIrsVDSCommand, log id: 2aa8f083 >2012-12-17 10:20:54,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4201b256] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c16cce5 >2012-12-17 10:20:54,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [4201b256] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:54,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [4201b256] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c16cce5 >2012-12-17 10:20:54,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:20:55,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5129 >2012-12-17 10:20:55,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5129, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 123ed952 >2012-12-17 10:20:55,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling started: taskId = bb7de44f-682e-4db6-961d-1315b85da3eb >2012-12-17 10:20:56,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended: taskId = bb7de44f-682e-4db6-961d-1315b85da3eb task status = finished >2012-12-17 10:20:56,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] spmStart polling ended, spm status: SPM >2012-12-17 10:20:56,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=bb7de44f-682e-4db6-961d-1315b85da3eb), log id: 8ff109a >2012-12-17 10:20:56,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMClearTaskVDSCommand, log id: 8ff109a >2012-12-17 10:20:56,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@219af313, log id: 123ed952 >2012-12-17 10:20:56,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:20:56,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:56,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 42ec0952 >2012-12-17 10:20:56,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42ec0952 >2012-12-17 10:20:56,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a768402 >2012-12-17 10:20:56,568 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:20:56,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 165808d1 >2012-12-17 10:20:56,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:20:56,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2135187 >2012-12-17 10:20:56,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2135187 >2012-12-17 10:20:56,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 165808d1 >2012-12-17 10:20:56,588 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-44) [19b7f9b6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:21:02,933 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [32133550] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:21:05,058 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [7646beaa] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:21:05,190 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [661fd7d8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:21:05,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 20982c93 >2012-12-17 10:21:05,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 336d0970 >2012-12-17 10:21:05,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:05,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, SpmStopVDSCommand, log id: 336d0970 >2012-12-17 10:21:05,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, ResetIrsVDSCommand, log id: 20982c93 >2012-12-17 10:21:05,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4bba8904 >2012-12-17 10:21:05,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:05,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4bba8904 >2012-12-17 10:21:13,231 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:21:15,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:21:15,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5130 >2012-12-17 10:21:15,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5130, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1762c3eb >2012-12-17 10:21:15,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling started: taskId = ac7841a2-3c41-439c-ad83-92f4266ca09c >2012-12-17 10:21:16,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling ended: taskId = ac7841a2-3c41-439c-ad83-92f4266ca09c task status = finished >2012-12-17 10:21:16,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] spmStart polling ended, spm status: SPM >2012-12-17 10:21:16,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e1a392] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ac7841a2-3c41-439c-ad83-92f4266ca09c), log id: 55c31f58 >2012-12-17 10:21:16,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, HSMClearTaskVDSCommand, log id: 55c31f58 >2012-12-17 10:21:16,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e1a392] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d3ef664, log id: 1762c3eb >2012-12-17 10:21:17,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:21:17,197 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:21:17,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 441096c4 >2012-12-17 10:21:17,237 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e1a392] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:21:17,429 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [77bd05d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:21:17,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 460f3232 >2012-12-17 10:21:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 6120de80 >2012-12-17 10:21:17,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:17,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, SpmStopVDSCommand, log id: 6120de80 >2012-12-17 10:21:17,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, ResetIrsVDSCommand, log id: 460f3232 >2012-12-17 10:21:17,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d28ec20 >2012-12-17 10:21:17,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [77bd05d1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:17,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [77bd05d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d28ec20 >2012-12-17 10:21:17,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:21:17,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5131 >2012-12-17 10:21:17,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5131, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 799ebab7 >2012-12-17 10:21:17,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling started: taskId = d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d >2012-12-17 10:21:18,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended: taskId = d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d task status = finished >2012-12-17 10:21:18,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] spmStart polling ended, spm status: SPM >2012-12-17 10:21:18,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=d90d8bc7-7767-4b7e-8b72-6b5a60e87d9d), log id: 6dbf493a >2012-12-17 10:21:19,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMClearTaskVDSCommand, log id: 6dbf493a >2012-12-17 10:21:19,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2960b720, log id: 799ebab7 >2012-12-17 10:21:19,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [3f5f0fad] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:21:19,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:19,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 6f7250a >2012-12-17 10:21:19,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f7250a >2012-12-17 10:21:19,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 441096c4 >2012-12-17 10:21:19,084 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:21:19,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2454af72 >2012-12-17 10:21:19,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:19,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 1c309d82 >2012-12-17 10:21:19,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c309d82 >2012-12-17 10:21:19,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-89) [3f5f0fad] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2454af72 >2012-12-17 10:21:19,104 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-89) [3f5f0fad] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:21:23,469 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [42fa3e4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:21:27,968 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [4020b36c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:21:28,142 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [7d28dac8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:21:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 433d12e6 >2012-12-17 10:21:28,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 31eb959c >2012-12-17 10:21:28,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:28,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, SpmStopVDSCommand, log id: 31eb959c >2012-12-17 10:21:28,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, ResetIrsVDSCommand, log id: 433d12e6 >2012-12-17 10:21:28,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6688c6d >2012-12-17 10:21:28,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [7d28dac8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:28,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [7d28dac8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6688c6d >2012-12-17 10:21:33,725 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:21:38,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:21:38,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5132 >2012-12-17 10:21:38,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5132, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e7b9d17 >2012-12-17 10:21:38,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling started: taskId = b827fcd2-a8f9-421b-ae06-53ef7ff56692 >2012-12-17 10:21:39,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended: taskId = b827fcd2-a8f9-421b-ae06-53ef7ff56692 task status = finished >2012-12-17 10:21:39,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] spmStart polling ended, spm status: SPM >2012-12-17 10:21:39,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=b827fcd2-a8f9-421b-ae06-53ef7ff56692), log id: 2078e1c7 >2012-12-17 10:21:39,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, HSMClearTaskVDSCommand, log id: 2078e1c7 >2012-12-17 10:21:39,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [26d9e89a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@602ee078, log id: 3e7b9d17 >2012-12-17 10:21:39,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:21:39,687 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:21:39,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 41a69a06 >2012-12-17 10:21:39,713 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [26d9e89a] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:21:39,908 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [499dcbe0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:21:40,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2eb0f4e0 >2012-12-17 10:21:40,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 723f44aa >2012-12-17 10:21:40,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:40,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, SpmStopVDSCommand, log id: 723f44aa >2012-12-17 10:21:40,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, ResetIrsVDSCommand, log id: 2eb0f4e0 >2012-12-17 10:21:40,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3992fffd >2012-12-17 10:21:40,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [499dcbe0] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:40,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [499dcbe0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3992fffd >2012-12-17 10:21:40,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:21:40,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5133 >2012-12-17 10:21:40,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5133, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fc42cc5 >2012-12-17 10:21:40,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling started: taskId = 54efb0ea-7fb2-4934-ae23-53bbf297c464 >2012-12-17 10:21:41,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling ended: taskId = 54efb0ea-7fb2-4934-ae23-53bbf297c464 task status = finished >2012-12-17 10:21:41,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] spmStart polling ended, spm status: SPM >2012-12-17 10:21:41,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=54efb0ea-7fb2-4934-ae23-53bbf297c464), log id: 4cf57c2f >2012-12-17 10:21:41,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMClearTaskVDSCommand, log id: 4cf57c2f >2012-12-17 10:21:41,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8ac1922, log id: 3fc42cc5 >2012-12-17 10:21:41,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:21:41,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:41,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 79b34030 >2012-12-17 10:21:41,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 79b34030 >2012-12-17 10:21:41,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41a69a06 >2012-12-17 10:21:41,601 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:21:41,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 673b4dbb >2012-12-17 10:21:41,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:41,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 3827853 >2012-12-17 10:21:41,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3827853 >2012-12-17 10:21:41,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 673b4dbb >2012-12-17 10:21:41,621 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-67) [1dd89be6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:21:44,007 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6c52cc2e] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:21:50,344 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [3c8f0aba] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:21:50,457 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [549524c7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:21:50,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 57b9a028 >2012-12-17 10:21:50,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f8c7ced >2012-12-17 10:21:50,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:50,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, SpmStopVDSCommand, log id: 4f8c7ced >2012-12-17 10:21:50,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, ResetIrsVDSCommand, log id: 57b9a028 >2012-12-17 10:21:50,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cc980ae >2012-12-17 10:21:50,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [549524c7] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:21:50,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cc980ae >2012-12-17 10:21:54,226 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:22:00,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:00,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5134 >2012-12-17 10:22:00,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5134, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ed2127b >2012-12-17 10:22:00,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling started: taskId = ab1ace5a-3589-4fb7-a050-84bba529d2c9 >2012-12-17 10:22:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling ended: taskId = ab1ace5a-3589-4fb7-a050-84bba529d2c9 task status = finished >2012-12-17 10:22:01,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] spmStart polling ended, spm status: SPM >2012-12-17 10:22:01,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=ab1ace5a-3589-4fb7-a050-84bba529d2c9), log id: 1bb7f8ee >2012-12-17 10:22:01,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, HSMClearTaskVDSCommand, log id: 1bb7f8ee >2012-12-17 10:22:01,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4b37df79, log id: 2ed2127b >2012-12-17 10:22:02,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:02,070 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:22:02,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1cae50b2 >2012-12-17 10:22:02,081 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [1f1f3d6e] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:02,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [692ca847] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:02,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 63353176 >2012-12-17 10:22:02,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 742a1311 >2012-12-17 10:22:02,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:02,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, SpmStopVDSCommand, log id: 742a1311 >2012-12-17 10:22:02,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, ResetIrsVDSCommand, log id: 63353176 >2012-12-17 10:22:02,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [692ca847] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24131433 >2012-12-17 10:22:02,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [692ca847] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:02,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [692ca847] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24131433 >2012-12-17 10:22:02,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:02,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5135 >2012-12-17 10:22:02,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5135, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73aa7313 >2012-12-17 10:22:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling started: taskId = dc148bb0-82c6-4180-bc19-36db42c4e1f9 >2012-12-17 10:22:03,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling ended: taskId = dc148bb0-82c6-4180-bc19-36db42c4e1f9 task status = finished >2012-12-17 10:22:03,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] spmStart polling ended, spm status: SPM >2012-12-17 10:22:03,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=dc148bb0-82c6-4180-bc19-36db42c4e1f9), log id: 75acaad3 >2012-12-17 10:22:03,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMClearTaskVDSCommand, log id: 75acaad3 >2012-12-17 10:22:03,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b033f00, log id: 73aa7313 >2012-12-17 10:22:03,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [b0e3bca] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:03,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:03,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: daaa0d0 >2012-12-17 10:22:03,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: daaa0d0 >2012-12-17 10:22:03,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1cae50b2 >2012-12-17 10:22:03,765 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:03,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3d23beb2 >2012-12-17 10:22:03,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:03,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 31dfa6f4 >2012-12-17 10:22:03,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31dfa6f4 >2012-12-17 10:22:03,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-43) [b0e3bca] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d23beb2 >2012-12-17 10:22:03,785 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-43) [b0e3bca] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:04,440 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [16841b99] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:22:12,633 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [729d4e04] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:12,756 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [5c7113d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:12,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 30f9728d >2012-12-17 10:22:12,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 5474c09d >2012-12-17 10:22:12,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:13,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, SpmStopVDSCommand, log id: 5474c09d >2012-12-17 10:22:13,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, ResetIrsVDSCommand, log id: 30f9728d >2012-12-17 10:22:13,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fba848d >2012-12-17 10:22:13,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [5c7113d1] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:13,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [5c7113d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fba848d >2012-12-17 10:22:14,680 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [406c9a80] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:22:23,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:23,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5136 >2012-12-17 10:22:23,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5136, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7539f35f >2012-12-17 10:22:23,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling started: taskId = 741a28ce-edb3-4672-b36d-af31fcbcb7b7 >2012-12-17 10:22:24,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling ended: taskId = 741a28ce-edb3-4672-b36d-af31fcbcb7b7 task status = finished >2012-12-17 10:22:24,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] spmStart polling ended, spm status: SPM >2012-12-17 10:22:24,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=741a28ce-edb3-4672-b36d-af31fcbcb7b7), log id: 62c5f928 >2012-12-17 10:22:24,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, HSMClearTaskVDSCommand, log id: 62c5f928 >2012-12-17 10:22:24,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [76c4364c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3368e3c1, log id: 7539f35f >2012-12-17 10:22:24,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:24,309 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:22:24,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43ee7ab1 >2012-12-17 10:22:24,324 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [76c4364c] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:24,427 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [56be0aeb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:24,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 25028416 >2012-12-17 10:22:24,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 17107575 >2012-12-17 10:22:24,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:24,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, SpmStopVDSCommand, log id: 17107575 >2012-12-17 10:22:24,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, ResetIrsVDSCommand, log id: 25028416 >2012-12-17 10:22:24,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7eaefa6c >2012-12-17 10:22:24,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [56be0aeb] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:24,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [56be0aeb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7eaefa6c >2012-12-17 10:22:24,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:24,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5137 >2012-12-17 10:22:24,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5137, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7809d515 >2012-12-17 10:22:24,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling started: taskId = f3e5a714-ea4c-4902-b133-38f0a98c247c >2012-12-17 10:22:25,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended: taskId = f3e5a714-ea4c-4902-b133-38f0a98c247c task status = finished >2012-12-17 10:22:25,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] spmStart polling ended, spm status: SPM >2012-12-17 10:22:25,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=f3e5a714-ea4c-4902-b133-38f0a98c247c), log id: 79e05bea >2012-12-17 10:22:25,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMClearTaskVDSCommand, log id: 79e05bea >2012-12-17 10:22:25,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78b29e28, log id: 7809d515 >2012-12-17 10:22:26,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [535051e4] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:26,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:26,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 323099d8 >2012-12-17 10:22:26,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 323099d8 >2012-12-17 10:22:26,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 43ee7ab1 >2012-12-17 10:22:26,668 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:26,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 14c44da6 >2012-12-17 10:22:26,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:26,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 51ebb28b >2012-12-17 10:22:26,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 51ebb28b >2012-12-17 10:22:26,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-85) [535051e4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 14c44da6 >2012-12-17 10:22:26,688 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-85) [535051e4] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:34,782 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [79d25c3b] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:34,882 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [d5f200c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:34,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6a750d19 >2012-12-17 10:22:34,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 7a7ed3d7 >2012-12-17 10:22:34,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:35,119 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [565ac2be] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:22:35,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, SpmStopVDSCommand, log id: 7a7ed3d7 >2012-12-17 10:22:35,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, ResetIrsVDSCommand, log id: 6a750d19 >2012-12-17 10:22:35,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fc2e3fb >2012-12-17 10:22:35,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [d5f200c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:35,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [d5f200c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fc2e3fb >2012-12-17 10:22:45,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:45,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5138 >2012-12-17 10:22:45,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5138, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11ede958 >2012-12-17 10:22:45,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling started: taskId = a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22 >2012-12-17 10:22:46,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended: taskId = a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22 task status = finished >2012-12-17 10:22:46,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] spmStart polling ended, spm status: SPM >2012-12-17 10:22:46,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=a1dde10b-3bb1-4d8c-ab63-d77d2c4ecf22), log id: 32eb168e >2012-12-17 10:22:46,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, HSMClearTaskVDSCommand, log id: 32eb168e >2012-12-17 10:22:46,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [34a366b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6815debf, log id: 11ede958 >2012-12-17 10:22:46,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:46,514 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:22:46,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9ed208b >2012-12-17 10:22:46,531 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [34a366b8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:46,687 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [5f08853c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:46,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 2319934b >2012-12-17 10:22:46,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 62da6a01 >2012-12-17 10:22:46,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:46,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, SpmStopVDSCommand, log id: 62da6a01 >2012-12-17 10:22:46,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, ResetIrsVDSCommand, log id: 2319934b >2012-12-17 10:22:46,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47ee91e8 >2012-12-17 10:22:46,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5f08853c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:46,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5f08853c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47ee91e8 >2012-12-17 10:22:47,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:22:47,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5139 >2012-12-17 10:22:47,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5139, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7295286 >2012-12-17 10:22:47,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling started: taskId = 5a65cda4-2a13-4c41-a23e-610495f75743 >2012-12-17 10:22:48,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling ended: taskId = 5a65cda4-2a13-4c41-a23e-610495f75743 task status = finished >2012-12-17 10:22:48,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] spmStart polling ended, spm status: SPM >2012-12-17 10:22:48,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=5a65cda4-2a13-4c41-a23e-610495f75743), log id: 42ea22ef >2012-12-17 10:22:48,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMClearTaskVDSCommand, log id: 42ea22ef >2012-12-17 10:22:48,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@199b9b3b, log id: 7295286 >2012-12-17 10:22:48,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60aea922] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:22:48,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:48,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 22a9285f >2012-12-17 10:22:48,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 22a9285f >2012-12-17 10:22:48,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 9ed208b >2012-12-17 10:22:48,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:48,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 285daf40 >2012-12-17 10:22:48,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:48,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 74c411b8 >2012-12-17 10:22:48,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 74c411b8 >2012-12-17 10:22:48,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-46) [60aea922] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 285daf40 >2012-12-17 10:22:48,239 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-46) [60aea922] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:22:55,564 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [3f24993] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:22:57,078 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [8102db] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:22:57,328 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [59abfc06] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:22:57,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: daf5a69 >2012-12-17 10:22:57,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2171236b >2012-12-17 10:22:57,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:57,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, SpmStopVDSCommand, log id: 2171236b >2012-12-17 10:22:57,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, ResetIrsVDSCommand, log id: daf5a69 >2012-12-17 10:22:57,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 262d12fa >2012-12-17 10:22:57,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [59abfc06] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:22:57,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [59abfc06] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 262d12fa >2012-12-17 10:23:03,705 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Checking if user admin@internal is an admin, result true >2012-12-17 10:23:03,706 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginAdminUserCommand internal: false. >2012-12-17 10:23:06,545 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68ede0fd] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:23:07,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:07,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5140 >2012-12-17 10:23:07,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5140, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b4ad868 >2012-12-17 10:23:07,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling started: taskId = 50cf4448-8014-4cf9-a98f-34e30f82b4e0 >2012-12-17 10:23:08,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended: taskId = 50cf4448-8014-4cf9-a98f-34e30f82b4e0 task status = finished >2012-12-17 10:23:08,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] spmStart polling ended, spm status: SPM >2012-12-17 10:23:08,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=50cf4448-8014-4cf9-a98f-34e30f82b4e0), log id: 37cdef84 >2012-12-17 10:23:08,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, HSMClearTaskVDSCommand, log id: 37cdef84 >2012-12-17 10:23:08,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) [1407e3a2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3127dfbb, log id: 1b4ad868 >2012-12-17 10:23:08,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:08,998 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:23:08,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28418e7e >2012-12-17 10:23:08,998 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [1407e3a2] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:23:09,112 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [49d26cb8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:23:09,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1e3042fa >2012-12-17 10:23:09,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 1f3563d3 >2012-12-17 10:23:09,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:09,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, SpmStopVDSCommand, log id: 1f3563d3 >2012-12-17 10:23:09,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, ResetIrsVDSCommand, log id: 1e3042fa >2012-12-17 10:23:09,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7220b4cc >2012-12-17 10:23:09,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [49d26cb8] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:09,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [49d26cb8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7220b4cc >2012-12-17 10:23:09,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:09,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5141 >2012-12-17 10:23:09,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5141, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f413fca >2012-12-17 10:23:09,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling started: taskId = 56283b57-fbe4-4f33-8767-0be7deea3156 >2012-12-17 10:23:10,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling ended: taskId = 56283b57-fbe4-4f33-8767-0be7deea3156 task status = finished >2012-12-17 10:23:10,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] spmStart polling ended, spm status: SPM >2012-12-17 10:23:10,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=56283b57-fbe4-4f33-8767-0be7deea3156), log id: 4e2b9f95 >2012-12-17 10:23:10,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMClearTaskVDSCommand, log id: 4e2b9f95 >2012-12-17 10:23:10,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11fa174b, log id: 1f413fca >2012-12-17 10:23:10,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [2b3eea84] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:10,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:10,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 75fa936f >2012-12-17 10:23:10,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 75fa936f >2012-12-17 10:23:10,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 28418e7e >2012-12-17 10:23:10,619 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:10,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15e598ec >2012-12-17 10:23:10,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:10,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 9ebd604 >2012-12-17 10:23:10,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9ebd604 >2012-12-17 10:23:10,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-29) [2b3eea84] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 15e598ec >2012-12-17 10:23:10,640 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-29) [2b3eea84] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:16,802 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [1dd89be6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:23:19,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [454eb5a9] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:23:19,669 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [4ba100c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:23:19,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 1a02fcd1 >2012-12-17 10:23:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 2c37892f >2012-12-17 10:23:19,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:19,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, SpmStopVDSCommand, log id: 2c37892f >2012-12-17 10:23:20,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, ResetIrsVDSCommand, log id: 1a02fcd1 >2012-12-17 10:23:20,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24634c5e >2012-12-17 10:23:20,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [4ba100c3] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:20,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [4ba100c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24634c5e >2012-12-17 10:23:27,077 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [102f58ae] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:23:30,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:30,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5142 >2012-12-17 10:23:30,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5142, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18f9e88d >2012-12-17 10:23:30,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling started: taskId = 1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea >2012-12-17 10:23:31,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling ended: taskId = 1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea task status = finished >2012-12-17 10:23:31,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] spmStart polling ended, spm status: SPM >2012-12-17 10:23:31,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=1dbd1b3d-bb9c-4539-8d4f-ae1d4196f8ea), log id: 4bcb844e >2012-12-17 10:23:31,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, HSMClearTaskVDSCommand, log id: 4bcb844e >2012-12-17 10:23:31,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) [661fd7d8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1e5b43c2, log id: 18f9e88d >2012-12-17 10:23:31,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:31,720 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:23:31,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27cb0408 >2012-12-17 10:23:31,783 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [661fd7d8] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:23:31,966 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [3118271c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:23:32,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 59d594b3 >2012-12-17 10:23:32,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 335a21ee >2012-12-17 10:23:32,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:32,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, SpmStopVDSCommand, log id: 335a21ee >2012-12-17 10:23:32,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, ResetIrsVDSCommand, log id: 59d594b3 >2012-12-17 10:23:32,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3118271c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 140aed25 >2012-12-17 10:23:32,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3118271c] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:32,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3118271c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 140aed25 >2012-12-17 10:23:32,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:32,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5143 >2012-12-17 10:23:32,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5143, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 798f9718 >2012-12-17 10:23:32,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling started: taskId = 70904064-a848-40ba-bb75-e754a51088fa >2012-12-17 10:23:33,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling ended: taskId = 70904064-a848-40ba-bb75-e754a51088fa task status = finished >2012-12-17 10:23:33,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] spmStart polling ended, spm status: SPM >2012-12-17 10:23:33,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=70904064-a848-40ba-bb75-e754a51088fa), log id: 558ac7f6 >2012-12-17 10:23:33,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMClearTaskVDSCommand, log id: 558ac7f6 >2012-12-17 10:23:33,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a4e135a, log id: 798f9718 >2012-12-17 10:23:33,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [65d7a0e6] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:33,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:33,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7eba84ab >2012-12-17 10:23:33,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7eba84ab >2012-12-17 10:23:33,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27cb0408 >2012-12-17 10:23:33,626 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:33,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 75c378d >2012-12-17 10:23:33,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:33,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 2440936 >2012-12-17 10:23:33,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2440936 >2012-12-17 10:23:33,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-14) [65d7a0e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75c378d >2012-12-17 10:23:33,648 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-14) [65d7a0e6] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:37,280 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [30ddd635] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:23:42,488 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [1290f5b5] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:23:42,776 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [24448a03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:23:42,840 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 6e36de98 >2012-12-17 10:23:42,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 441d3e65 >2012-12-17 10:23:42,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:42,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, SpmStopVDSCommand, log id: 441d3e65 >2012-12-17 10:23:43,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, ResetIrsVDSCommand, log id: 6e36de98 >2012-12-17 10:23:43,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [24448a03] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6bdfa4c6 >2012-12-17 10:23:43,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [24448a03] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:43,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [24448a03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6bdfa4c6 >2012-12-17 10:23:47,693 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [681720a4] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:23:53,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:53,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5144 >2012-12-17 10:23:53,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5144, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1c10d5b8 >2012-12-17 10:23:53,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling started: taskId = 66bda7d6-1474-437f-bb59-02b7b2b420cf >2012-12-17 10:23:54,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling ended: taskId = 66bda7d6-1474-437f-bb59-02b7b2b420cf task status = finished >2012-12-17 10:23:54,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] spmStart polling ended, spm status: SPM >2012-12-17 10:23:54,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=66bda7d6-1474-437f-bb59-02b7b2b420cf), log id: 4ccc650d >2012-12-17 10:23:54,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, HSMClearTaskVDSCommand, log id: 4ccc650d >2012-12-17 10:23:54,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [7aeab1da] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7efe5fde, log id: 1c10d5b8 >2012-12-17 10:23:54,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:54,423 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Attempting to get and stop tasks on storage pool TEST-DC >2012-12-17 10:23:54,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a4d6a62 >2012-12-17 10:23:54,439 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [7aeab1da] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:23:54,672 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [2144ca30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:23:54,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 667b98ad >2012-12-17 10:23:54,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 67111f83 >2012-12-17 10:23:54,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:54,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, SpmStopVDSCommand, log id: 67111f83 >2012-12-17 10:23:54,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, ResetIrsVDSCommand, log id: 667b98ad >2012-12-17 10:23:54,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3083d35c >2012-12-17 10:23:54,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [2144ca30] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:54,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [2144ca30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3083d35c >2012-12-17 10:23:54,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] hostFromVds::selectedVds - pink-vds2, spmStatus Free, storage pool TEST-DC >2012-12-17 10:23:55,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] starting spm on vds pink-vds2, storage pool TEST-DC, prevId -1, LVER 5145 >2012-12-17 10:23:55,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SpmStartVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, prevId=-1, prevLVER=5145, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e329617 >2012-12-17 10:23:55,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling started: taskId = 2b0eee60-ab04-49a7-aeec-7ac50dc3b87c >2012-12-17 10:23:56,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended: taskId = 2b0eee60-ab04-49a7-aeec-7ac50dc3b87c task status = finished >2012-12-17 10:23:56,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] spmStart polling ended, spm status: SPM >2012-12-17 10:23:56,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMClearTaskVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, taskId=2b0eee60-ab04-49a7-aeec-7ac50dc3b87c), log id: 33183034 >2012-12-17 10:23:56,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMClearTaskVDSCommand, log id: 33183034 >2012-12-17 10:23:56,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7904c020, log id: 7e329617 >2012-12-17 10:23:56,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [c06c0c1] Initialize Irs proxy from vds: pink-vds2.qa.lab.tlv.redhat.com >2012-12-17 10:23:56,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:56,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 7ce42015 >2012-12-17 10:23:56,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7ce42015 >2012-12-17 10:23:56,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4a4d6a62 >2012-12-17 10:23:56,886 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:56,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 504e9ca9 >2012-12-17 10:23:56,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:23:56,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] START, HSMGetAllTasksInfoVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455), log id: 685dcd3f >2012-12-17 10:23:56,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 685dcd3f >2012-12-17 10:23:56,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-21) [c06c0c1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 504e9ca9 >2012-12-17 10:23:56,907 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-21) [c06c0c1] Discovered no tasks on Storage Pool TEST-DC >2012-12-17 10:23:58,337 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [36ca77a1] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:24:05,017 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [b18376] Master domain version is not in sync between DB and VDSM. Domain st marked as master, but the version in DB: 1 and in VDSM: 2 >2012-12-17 10:24:05,119 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [7f5c814d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 2863bd4d-3f70-4ae8-b186-bf58088e28bd Type: Storage >2012-12-17 10:24:05,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, ResetIrsVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = pink-vds2.qa.lab.tlv.redhat.com, vdsId = 8ed2188e-4795-11e2-b963-001122334455, ignoreStopFailed = true), log id: 69aeca7c >2012-12-17 10:24:05,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, SpmStopVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd), log id: 4f07adff >2012-12-17 10:24:05,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] SpmStopVDSCommand::Stopping SPM on vds pink-vds2, pool id 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:24:05,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, SpmStopVDSCommand, log id: 4f07adff >2012-12-17 10:24:05,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, ResetIrsVDSCommand, log id: 69aeca7c >2012-12-17 10:24:05,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e52a2e2 >2012-12-17 10:24:05,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7f5c814d] clearing cache for problematic entities in pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd >2012-12-17 10:24:05,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7f5c814d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e52a2e2 >2012-12-17 10:24:08,559 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [19b7f9b6] Storage Domain 2863bd4d-3f70-4ae8-b186-bf58088e28bd:st was reported by Host pink-vds2 as Active in Pool 03bf4661-cd9b-4581-a18f-ed7f7be148dd, moving to active status >2012-12-17 10:24:11,871 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-50) [46708fc2] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 8ed2188e-4795-11e2-b963-001122334455 Type: VDS >2012-12-17 10:24:11,873 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [46708fc2] START, SetVdsStatusVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 2f9d8b07 >2012-12-17 10:24:11,916 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [46708fc2] FINISH, SetVdsStatusVDSCommand, log id: 2f9d8b07 >2012-12-17 10:24:12,002 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-50) [46708fc2] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 8ed2188e-4795-11e2-b963-001122334455 Type: VDS >2012-12-17 10:24:12,636 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [4007c40f] vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 8ed2188e-4795-11e2-b963-001122334455 : pink-vds2 >2012-12-17 10:24:12,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [4007c40f] Clearing cache of pool: 03bf4661-cd9b-4581-a18f-ed7f7be148dd for problematic entities of VDS: pink-vds2. >2012-12-17 10:24:12,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] START, DisconnectStoragePoolVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, vds_spm_id = 1), log id: 73fd3692 >2012-12-17 10:24:14,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [4007c40f] FINISH, DisconnectStoragePoolVDSCommand, log id: 73fd3692 >2012-12-17 10:24:14,825 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (QuartzScheduler_Worker-55) [71356416] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 03bf4661-cd9b-4581-a18f-ed7f7be148dd Type: StoragePool >2012-12-17 10:24:14,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [71356416] START, DisconnectStorageServerVDSCommand(HostName = pink-vds2, HostId = 8ed2188e-4795-11e2-b963-001122334455, storagePoolId = 03bf4661-cd9b-4581-a18f-ed7f7be148dd, storageType = NFS, connectionList = [{ id: e6b2daa0-82fe-4c87-af44-a7017d09cdca, connection: orion.qa.lab.tlv.redhat.com:/export/gena/st_3000, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 9d5d0cd >2012-12-17 10:24:18,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [71356416] FINISH, DisconnectStorageServerVDSCommand, return: {e6b2daa0-82fe-4c87-af44-a7017d09cdca=0}, log id: 9d5d0cd >2012-12-17 10:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Checking autorecoverable hosts >2012-12-17 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Autorecovering 0 hosts >2012-12-17 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Checking autorecoverable hosts done >2012-12-17 10:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50119c91] Autorecovering storage domains is disabled, skipping >2012-12-17 10:25:51,779 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-58) [5cdf4906] DbUserCacheManager::refreshAllUserData() - entered >2012-12-17 10:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Checking autorecoverable hosts >2012-12-17 10:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Autorecovering 0 hosts >2012-12-17 10:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Checking autorecoverable hosts done >2012-12-17 10:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [53480d94] Autorecovering storage domains is disabled, skipping >2012-12-17 10:34:15,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [4fc51f8f] vds::refreshVdsStats Failed getVdsStats, vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, error = VDSNetworkException: VDSNetworkException: >2012-12-17 10:34:15,636 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-19) [4fc51f8f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing. >VDSNetworkException: >2012-12-17 10:34:17,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) [24cb4742] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing. >VDSNetworkException: >2012-12-17 10:34:19,652 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-8) [536ede7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 39a7379e-439a-11e2-ba16-001122334455 : orchid-vds2, VDS Network Error, continuing. >VDSNetworkException: >2012-12-17 10:34:20,805 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-20) [76ee6c44] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:34:20,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30 >2012-12-17 10:34:20,936 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] SPM Init: could not find reported vds or not up - pool:CL30 vds_spm_id: 2 >2012-12-17 10:34:20,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] SPM selection - vds seems as spm orchid-vds2 >2012-12-17 10:34:20,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] START, SpmStopVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96), log id: 76cc1ce6 >2012-12-17 10:34:20,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] SpmStopVDSCommand:: vds orchid-vds2 is in Connecting status - not performing spm stop, pool id 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:34:20,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-20) [76ee6c44] FINISH, SpmStopVDSCommand, log id: 76cc1ce6 >2012-12-17 10:34:20,939 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [76ee6c44] spm stop on spm failed, stopping spm selection! >2012-12-17 10:34:22,703 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [7b7a7712] Running command: InitVdsOnUpCommand internal: true. >2012-12-17 10:34:22,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ValidateStorageServerConnectionVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 662c11c0 >2012-12-17 10:34:22,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ValidateStorageServerConnectionVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 662c11c0 >2012-12-17 10:34:22,812 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [4464e415] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:34:22,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ConnectStorageServerVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4263722c >2012-12-17 10:34:22,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ConnectStorageServerVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 4263722c >2012-12-17 10:34:22,835 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [4464e415] Host orchid-vds2 storage connection was succeeded >2012-12-17 10:34:22,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) [4464e415] START, ConnectStoragePoolVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, vds_spm_id = 2, masterDomainId = e56c6a97-fb9a-4e87-b77b-046bd7f29be0, masterVersion = 1), log id: 273d2fd7 >2012-12-17 10:34:28,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) [4464e415] FINISH, ConnectStoragePoolVDSCommand, log id: 273d2fd7 >2012-12-17 10:34:28,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [4464e415] No string for UNASSIGNED type. Use default Log >2012-12-17 10:34:28,177 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) [4464e415] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-12-17 10:34:28,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [4ee87fc2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 39a7379e-439a-11e2-ba16-001122334455 Type: VDS >2012-12-17 10:34:28,227 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [7b61ca13] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 39a7379e-439a-11e2-ba16-001122334455 Type: VDS >2012-12-17 10:34:30,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30 >2012-12-17 10:34:30,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Found spm host 39a7379e-439a-11e2-ba16-001122334455, host name: orchid-vds2, according to spmId: 2. >2012-12-17 10:34:30,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] SpmStatus on vds 39a7379e-439a-11e2-ba16-001122334455: Free >2012-12-17 10:34:31,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] starting spm on vds orchid-vds1, storage pool CL30, prevId 2, LVER 0 >2012-12-17 10:34:31,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, SpmStartVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, prevId=2, prevLVER=0, storagePoolFormatType=V1, recoveryMode=Manual, SCSIFencing=false), log id: 2db22385 >2012-12-17 10:34:31,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling started: taskId = 98e27a6d-d465-4e7e-8607-726b183f76ac >2012-12-17 10:34:53,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling ended: taskId = 98e27a6d-d465-4e7e-8607-726b183f76ac task status = finished >2012-12-17 10:34:53,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] spmStart polling ended, spm status: SPM >2012-12-17 10:34:53,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] START, HSMClearTaskVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, taskId=98e27a6d-d465-4e7e-8607-726b183f76ac), log id: 49a8a30 >2012-12-17 10:34:53,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, HSMClearTaskVDSCommand, log id: 49a8a30 >2012-12-17 10:34:53,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [2d5b99a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3de657ac, log id: 2db22385 >2012-12-17 10:34:53,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [2d5b99a4] Initialize Irs proxy from vds: orchid-vds1.qa.lab.tlv.redhat.com >2012-12-17 10:34:53,969 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Attempting to get and stop tasks on storage pool CL30 >2012-12-17 10:34:53,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 324aa7e6 >2012-12-17 10:34:54,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:34:54,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 5622bea0 >2012-12-17 10:34:54,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5622bea0 >2012-12-17 10:34:54,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 324aa7e6 >2012-12-17 10:34:54,068 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Discovered no tasks on Storage Pool CL30 >2012-12-17 10:34:54,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 311268e >2012-12-17 10:34:54,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:34:54,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 703af56a >2012-12-17 10:34:54,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 703af56a >2012-12-17 10:34:54,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) [565ac2be] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 311268e >2012-12-17 10:34:54,075 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) [565ac2be] Discovered no tasks on Storage Pool CL30 >2012-12-17 10:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Checking autorecoverable hosts >2012-12-17 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Autorecovering 0 hosts >2012-12-17 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Checking autorecoverable hosts done >2012-12-17 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [57fa1ec] Autorecovering storage domains is disabled, skipping >2012-12-17 10:36:00,109 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-3-thread-48) [e712868] Running command: MigrateVmCommand internal: false. Entities affected : ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM >2012-12-17 10:36:00,120 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-48) [e712868] START, MigrateVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstVdsId=39a7379e-439a-11e2-ba16-001122334455, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 7bb4865f >2012-12-17 10:36:00,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] VdsBroker::migrate::Entered (vm_guid=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321, method=online >2012-12-17 10:36:00,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] START, MigrateBrokerVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds1.qa.lab.tlv.redhat.com, dstVdsId=39a7379e-439a-11e2-ba16-001122334455, dstHost=orchid-vds2.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 41afcf86 >2012-12-17 10:36:00,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-48) [e712868] FINISH, MigrateBrokerVDSCommand, log id: 41afcf86 >2012-12-17 10:36:00,164 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-48) [e712868] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7bb4865f >2012-12-17 10:36:00,574 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [76ee6c44] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:02,627 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [3d5285e1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:04,678 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [956a2f9] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:06,900 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [5c1b0caf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:08,945 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [5c7113d1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:10,987 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [406c9a80] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:13,032 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [77748919] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:15,080 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [127fd955] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:17,211 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) [2b3eea84] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:19,255 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [56be0aeb] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:21,295 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [25e2aebf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:23,340 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [499dcbe0] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:25,388 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [7f5c814d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:27,495 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [50119c91] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:29,539 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [d5f200c] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:31,581 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [1630089b] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:36:33,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-88) [549524c7] START, DestroyVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, force=false, secondsToWait=0, gracefully=false), log id: 420b884a >2012-12-17 10:36:33,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-88) [549524c7] FINISH, DestroyVDSCommand, log id: 420b884a >2012-12-17 10:36:33,512 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [549524c7] Setting VM vm_30 e9e35b3d-031d-4ef3-a8b4-4fd83936b75b to status unknown >2012-12-17 10:36:33,512 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [549524c7] vm vm_30 running in db and not running in vds - add to rerun treatment. vds orchid-vds1 >2012-12-17 10:37:52,032 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-64) [77748919] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 7d08a922-3d3d-11e2-82f1-001122334455 : orchid-vds1, VDS Network Error, continuing. >VDSNetworkException: >2012-12-17 10:37:54,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-12) [49d26cb8] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 7d08a922-3d3d-11e2-82f1-001122334455 : orchid-vds1, VDS Network Error, continuing. >VDSNetworkException: >2012-12-17 10:37:55,791 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-28) [3484e0f5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:37:55,850 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-12-17 10:37:55,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] Irs placed on server null failed. Proceed Failover >2012-12-17 10:37:55,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] hostFromVds::selectedVds - orchid-vds2, spmStatus Free, storage pool CL30 >2012-12-17 10:37:55,901 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] SPM Init: could not find reported vds or not up - pool:CL30 vds_spm_id: 1 >2012-12-17 10:37:55,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] SPM selection - vds seems as spm orchid-vds1 >2012-12-17 10:37:55,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] START, SpmStopVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96), log id: 4cef7774 >2012-12-17 10:37:55,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] SpmStopVDSCommand:: vds orchid-vds1 is in Connecting status - not performing spm stop, pool id 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:37:55,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-28) [3484e0f5] FINISH, SpmStopVDSCommand, log id: 4cef7774 >2012-12-17 10:37:55,904 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [3484e0f5] spm stop on spm failed, stopping spm selection! >2012-12-17 10:37:57,123 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-29) [221bb562] Running command: InitVdsOnUpCommand internal: true. >2012-12-17 10:37:57,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ValidateStorageServerConnectionVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 51e4d110 >2012-12-17 10:37:57,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ValidateStorageServerConnectionVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 51e4d110 >2012-12-17 10:37:57,227 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [58834ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:37:57,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ConnectStorageServerVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, storageType = NFS, connectionList = [{ id: 696c5b0d-330e-4667-89c7-428563d6fc47, connection: orion.qa.lab.tlv.redhat.com:/export/meni/upgrade, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 22c5ae7d >2012-12-17 10:37:57,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ConnectStorageServerVDSCommand, return: {696c5b0d-330e-4667-89c7-428563d6fc47=0}, log id: 22c5ae7d >2012-12-17 10:37:57,250 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [58834ae] Host orchid-vds1 storage connection was succeeded >2012-12-17 10:37:57,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-29) [58834ae] START, ConnectStoragePoolVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, vds_spm_id = 1, masterDomainId = e56c6a97-fb9a-4e87-b77b-046bd7f29be0, masterVersion = 1), log id: 7b258fe >2012-12-17 10:38:03,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-29) [58834ae] FINISH, ConnectStoragePoolVDSCommand, log id: 7b258fe >2012-12-17 10:38:03,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [58834ae] No string for UNASSIGNED type. Use default Log >2012-12-17 10:38:03,983 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) [58834ae] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-12-17 10:38:04,015 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [3b77614b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS >2012-12-17 10:38:04,032 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [dbfa1dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS >2012-12-17 10:38:05,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] hostFromVds::selectedVds - orchid-vds1, spmStatus Free, storage pool CL30 >2012-12-17 10:38:05,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] SpmStatus on vds 7d08a922-3d3d-11e2-82f1-001122334455: Free >2012-12-17 10:38:06,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] starting spm on vds orchid-vds1, storage pool CL30, prevId 1, LVER 1 >2012-12-17 10:38:06,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, SpmStartVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, prevId=1, prevLVER=1, storagePoolFormatType=V1, recoveryMode=Manual, SCSIFencing=false), log id: 1b7ee2fe >2012-12-17 10:38:06,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling started: taskId = 2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf >2012-12-17 10:38:31,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling ended: taskId = 2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf task status = finished >2012-12-17 10:38:31,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] spmStart polling ended, spm status: SPM >2012-12-17 10:38:31,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] START, HSMClearTaskVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, taskId=2e0b6e5a-ce98-4b8c-81fb-92766f26e5cf), log id: 7efde103 >2012-12-17 10:38:31,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, HSMClearTaskVDSCommand, log id: 7efde103 >2012-12-17 10:38:31,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [5ee48710] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dbfcd17, log id: 1b7ee2fe >2012-12-17 10:38:32,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [5ee48710] Initialize Irs proxy from vds: orchid-vds1.qa.lab.tlv.redhat.com >2012-12-17 10:38:32,254 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Attempting to get and stop tasks on storage pool CL30 >2012-12-17 10:38:32,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 459c68b8 >2012-12-17 10:38:32,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:38:32,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 6d235702 >2012-12-17 10:38:32,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d235702 >2012-12-17 10:38:32,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 459c68b8 >2012-12-17 10:38:32,357 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Discovered no tasks on Storage Pool CL30 >2012-12-17 10:38:32,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 27ecc06e-1f3c-4470-ac7d-94740ed00e96, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2b257c48 >2012-12-17 10:38:32,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 27ecc06e-1f3c-4470-ac7d-94740ed00e96 >2012-12-17 10:38:32,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] START, HSMGetAllTasksInfoVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455), log id: 7440eaf0 >2012-12-17 10:38:32,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7440eaf0 >2012-12-17 10:38:32,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-98) [780047c3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b257c48 >2012-12-17 10:38:32,376 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) [780047c3] Discovered no tasks on Storage Pool CL30 >2012-12-17 10:38:34,420 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-3-thread-50) [12a74ae8] Running command: MigrateVmCommand internal: false. Entities affected : ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM >2012-12-17 10:38:34,432 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-50) [12a74ae8] START, MigrateVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstVdsId=7d08a922-3d3d-11e2-82f1-001122334455, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 460b55c6 >2012-12-17 10:38:34,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] VdsBroker::migrate::Entered (vm_guid=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321, method=online >2012-12-17 10:38:34,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] START, MigrateBrokerVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, srcHost=orchid-vds2.qa.lab.tlv.redhat.com, dstVdsId=7d08a922-3d3d-11e2-82f1-001122334455, dstHost=orchid-vds1.qa.lab.tlv.redhat.com:54321, migrationMethod=ONLINE), log id: 795d25b2 >2012-12-17 10:38:34,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-3-thread-50) [12a74ae8] FINISH, MigrateBrokerVDSCommand, log id: 795d25b2 >2012-12-17 10:38:34,487 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-3-thread-50) [12a74ae8] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 460b55c6 >2012-12-17 10:38:34,888 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [515f648d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:36,951 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [59abfc06] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:39,000 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [7e43b5c5] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:41,121 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [5c1b0caf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:43,161 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [2d67b14a] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:45,202 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [2144ca30] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:47,245 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [3484e0f5] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:49,296 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [7dff3777] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:51,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [742a6686] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:53,450 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [499dcbe0] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:55,498 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [15d070bf] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:57,539 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [2c0da98a] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:38:59,589 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [4d8dc1c4] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:01,913 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [565ac2be] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:03,958 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [5cdf4906] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:06,003 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [53480d94] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:08,047 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [6018e3] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:10,098 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [10ed878d] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:12,219 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [5bc6d47] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:14,266 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [18826afe] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:16,308 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [60aea922] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:18,351 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-30) [533115ff] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:20,401 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [77bd05d1] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b is migrating to vds orchid-vds1 ignoring it in the refresh until migration is done >2012-12-17 10:39:24,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-4) [3f24993] START, DestroyVDSCommand(HostName = orchid-vds2, HostId = 39a7379e-439a-11e2-ba16-001122334455, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, force=false, secondsToWait=0, gracefully=false), log id: 224d20c6 >2012-12-17 10:39:24,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-4) [3f24993] FINISH, DestroyVDSCommand, log id: 224d20c6 >2012-12-17 10:39:24,126 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [3f24993] vds::refreshVmList vm id e9e35b3d-031d-4ef3-a8b4-4fd83936b75b status = Down on vds orchid-vds2 ignoring it in the refresh until migration is done >2012-12-17 10:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Checking autorecoverable hosts >2012-12-17 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Autorecovering 0 hosts >2012-12-17 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Checking autorecoverable hosts done >2012-12-17 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [3118271c] Autorecovering storage domains is disabled, skipping >2012-12-17 10:40:08,385 INFO [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-13) [12e20649] Running command: AddNetworkCommand internal: false. Entities affected : ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:40:08,492 INFO [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [4e622b49] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected : ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups >2012-12-17 10:40:18,663 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 2/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,666 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 3/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,670 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-48) Start time: 12/17/12 10:40 AM. Start running CanDoAction for command number 1/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,678 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 3/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,687 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-48) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 1/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,689 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) End time: 12/17/12 10:40 AM. Finish handling CanDoAction for command number 2/3 (Command type: AttachNetworkToVdsGroup) >2012-12-17 10:40:18,690 INFO [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [87ccf1e] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected : ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups >2012-12-17 10:40:18,752 INFO [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [3a01beec] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected : ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups >2012-12-17 10:40:18,819 INFO [org.ovirt.engine.core.bll.AttachNetworkToVdsGroupCommand] (pool-3-thread-50) [489d95b4] Running command: AttachNetworkToVdsGroupCommand internal: false. Entities affected : ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups >2012-12-17 10:41:24,356 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-2) [6d9c714c] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: b3bc9dd0-4787-11e2-afa9-001122334455 Type: VdsGroups >2012-12-17 10:41:29,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) [19b7f9b6] START, FullListVdsCommand( HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vds=null, vmIds=[e9e35b3d-031d-4ef3-a8b4-4fd83936b75b]), log id: 3f48edde >2012-12-17 10:41:29,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) [19b7f9b6] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@4cf8aa47, log id: 3f48edde >2012-12-17 10:41:29,893 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [19b7f9b6] Recieved a memballoon Device without an address when processing VM e9e35b3d-031d-4ef3-a8b4-4fd83936b75b devices, skipping device: {specParams={model=none}, device=memballoon, type=balloon} >2012-12-17 10:42:13,764 INFO [org.ovirt.engine.core.bll.SetupNetworksCommand] (ajp-/127.0.0.1:8702-5) [530f614] Running command: SetupNetworksCommand internal: false. Entities affected : ID: 7d08a922-3d3d-11e2-82f1-001122334455 Type: VDS >2012-12-17 10:42:13,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] START, SetupNetworksVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[vlan_222 {id=ed425f9f-a400-46c8-8b78-625512979038, description=null, subnet=null, gateway=null, type=null, vlan_id=222, stp=false, storage_pool_id=27ecc06e-1f3c-4470-ac7d-94740ed00e96, mtu=0, vmNetwork=true, cluster=network_cluster {id={clusterId=null, networkId=null}, status=Operational, is_display=false, required=false}}], > bonds=[], > interfaces=[eth1.111 {id=cd8d98a8-ca02-4cdd-a0e2-a04884a118b8, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth1.111, networkName=vlan111, vlanId=111, bootProtocol=StaticIp, address=2.2.2.2, subnet=255.255.255.255, gateway=null, mtu=9000, bridged=true, type=0, networkImplementationDetails={inSync=true, managed=true}}, > bond0 {id=cd52dd54-1356-4381-8df5-a2cb60de8246, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=bond0, macAddress=00:14:5E:17:D5:B0, networkName=vmnet1, bondOptions=mode=1 miimon=100, bootProtocol=StaticIp, address=1.1.1.1, subnet=255.255.255.255, gateway=null, mtu=9000, bridged=true, type=0, networkImplementationDetails={inSync=true, managed=true}}, > eth3 {id=ffa6db0e-234b-4ebc-96aa-75b59859a649, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth3, macAddress=00:14:5E:17:D5:B2, networkName=null, bondName=bond0, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, > eth1 {id=cdcf9e00-644e-4e2d-91b4-1d28fe0b3740, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth1, macAddress=00:15:17:3D:CD:AB, networkName=null, bondName=null, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, > eth2 {id=d35e2d5e-3a6e-4ff7-8529-8f7c5419ebb2, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth2, macAddress=00:14:5E:17:D5:B0, networkName=null, bondName=bond0, bootProtocol=None, address=, subnet=, gateway=null, mtu=9000, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, > eth0 {id=551bc1b8-a110-4d28-97fd-f6f6f4f02b1f, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, name=eth0, macAddress=00:15:17:3D:CD:AA, networkName=rhevm, bondName=null, bootProtocol=Dhcp, address=10.35.97.58, subnet=255.255.255.0, gateway=10.35.97.254, mtu=1500, bridged=true, speed=1000, type=2, networkImplementationDetails={inSync=true, managed=true}}, > eth1.222 {id=null, vdsId=7d08a922-3d3d-11e2-82f1-001122334455, macAddress=null, networkName=vlan_222, vlanId=222, bonded=null, bondName=null, bondOptions=null, bootProtocol=Dhcp, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=null, type=0, networkImplementationDetails=null}], > removedNetworks=[], > removedBonds=[]), log id: 7a309a12 >2012-12-17 10:42:13,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] FINISH, SetupNetworksVDSCommand, log id: 7a309a12 >2012-12-17 10:42:16,668 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] START, UpdateVdsDynamicDataVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@bd34306a), log id: 356d244c >2012-12-17 10:42:16,669 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (ajp-/127.0.0.1:8702-5) [530f614] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 356d244c >2012-12-17 10:42:17,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [530f614] No string for UNASSIGNED type. Use default Log >2012-12-17 10:42:35,556 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) [4fd9154] Adding new value to configuration cache. >2012-12-17 10:42:35,557 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) [4fd9154] Didnt find the value of MaxMTU in DB for version general - using default: 9,000 >2012-12-17 10:42:35,558 WARN [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-7) [4fd9154] CanDoAction of action AddNetwork failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__NETWORK,NETWORK_MTU_OVERRIDE_NOT_SUPPORTED >2012-12-17 10:43:09,983 WARN [org.ovirt.engine.core.bll.storage.AddNetworkCommand] (ajp-/127.0.0.1:8702-2) [2cc9fe09] CanDoAction of action AddNetwork failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__NETWORK,NETWORK_MTU_OVERRIDE_NOT_SUPPORTED >2012-12-17 10:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Checking autorecoverable hosts >2012-12-17 10:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Autorecovering 0 hosts >2012-12-17 10:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Checking autorecoverable hosts done >2012-12-17 10:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1ecb2997] Autorecovering storage domains is disabled, skipping >2012-12-17 10:47:16,410 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-8) [2c86b147] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM, ID: 27ecc06e-1f3c-4470-ac7d-94740ed00e96 Type: StoragePool >2012-12-17 10:47:16,412 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [2c86b147] MacPoolManager::allocateNewMac entered >2012-12-17 10:47:16,412 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [2c86b147] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:16:88:59 >2012-12-17 10:47:17,473 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-8) [2f05496] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: e9e35b3d-031d-4ef3-a8b4-4fd83936b75b Type: VM >2012-12-17 10:47:17,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-8) [2f05496] START, HotPlugNicVDSCommand(HostName = orchid-vds1, HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vm.vm_name=vm_30, nic=VmNetworkInterface {id=c606f989-1e33-45c6-bfd1-69bd9fa38a02, networkName=vlan_222, speed=1000, type=3, name=nic4, macAddress=00:1a:4a:16:88:59, active=true, portMirroring=true, vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, vmName=vm_30, vmTemplateId=null}, vmDevice=VmDevice {vmId=e9e35b3d-031d-4ef3-a8b4-4fd83936b75b, deviceId=c606f989-1e33-45c6-bfd1-69bd9fa38a02, device=bridge, type=interface, bootOrder=0, specParams={}, address=, managed=true, plugged=true, readOnly=false, alias=}), log id: 6bdd41b4 >2012-12-17 10:47:18,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-8) [2f05496] FINISH, HotPlugNicVDSCommand, log id: 6bdd41b4 >2012-12-17 10:47:24,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-67) [1dd89be6] START, FullListVdsCommand( HostId = 7d08a922-3d3d-11e2-82f1-001122334455, vds=null, vmIds=[e9e35b3d-031d-4ef3-a8b4-4fd83936b75b]), log id: 3c4dfa0f >2012-12-17 10:47:24,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-67) [1dd89be6] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@235c0db7, log id: 3c4dfa0f >2012-12-17 10:47:24,101 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [1dd89be6] Recieved a memballoon Device without an address when processing VM e9e35b3d-031d-4ef3-a8b4-4fd83936b75b devices, skipping device: {specParams={model=none}, device=memballoon, type=balloon} >2012-12-17 10:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Checking autorecoverable hosts >2012-12-17 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Autorecovering 0 hosts >2012-12-17 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Checking autorecoverable hosts done >2012-12-17 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [19b7f9b6] Autorecovering storage domains is disabled, skipping
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 887751
: 664728